VM creation fails because lvmlockd leaks VG shared locks after storage disconnects

Environment

• Product Line: ZCF
• Versions: Unspecified
• Components: ZStack Cloud
• Trigger Condition: Shared storage clusters utilizing lvmlockd (typically SharedBlock or SAN-backed primary storage deployments in ZStack).

Symptoms

• A temporary disconnection between storage and hosts causes unreleased Volume Group (VG) shared locks (sh lock) to remain in lvmlockd.
• Because creating a new volume requires acquiring a VG exclusive lock (ex lock), subsequent operations (such as running vgs or lvchange) leak additional sh locks each time.
• This results in a deadlock that causes VM creation requests to fail.

Diagnosis

1. Execute vgs on the host. The command freezes if the VG lock is trapped.
2. Execute vgs --no-locking on the host. If it returns results immediately, it confirms the bottleneck is caused by the lock manager (lvmlockd) rather than underlying storage hardware.
3. Execute dmesg -T | grep -i error to confirm whether repeating I/O error messages appeared around the time VM creation began failing.

Root cause

• Temporary disconnections between host and storage cause residual VG sh locks to remain in lvmlockd, and automatic recovery processes fail to clear them.
• Hardware-level faults (e.g., optical module/port failures and fiber cable issues) cause recurring I/O errors, which lead to continuous lock leakage.
• Stale or stuck tasks in the Management Node task queue retain stale lock references, blocking subsequent volume creation requests.

Resolution

1. Confirm the VG lock bottleneck

• Run the following on the host:

  vgs

• Symptom Confirmation: If locked, expected behavior is that the command freezes.
• Verify with command below to confirm lock contention:

  vgs --no-locking

• Expected result: vgs --no-locking returns immediately, confirming the bottleneck is caused by the lock manager rather than underlying storage hardware.

2. Confirm the recent storage disconnections

• Run command:

  dmesg -T | grep -i error

• Expected result: Verify that storage link interruptions triggered the sh lock leakage.

3. Restart lock services on affected compute nodes

• Run command:

  systemctl restart sanlock
  systemctl restart lvm2-lvmlockd

• Check for residual locks:

  sanlock client status | grep VGLK

• Exception Handling: If this count continues to increase after restarting compute nodes, proceed to Step 4.

4. Eliminate storage-side IO errors

• Continue monitoring dmesg -T | grep -i error. If IO error continues to recur, lock leaks will happen again.
• Action: Locate the affected storage port (typically suspect HBA cards / fiber channel / optical modules on the host) and replace them.

5. Restart lock services on the management node

• Run on the management node responsible for the cluster:

  systemctl restart sanlock
  systemctl restart lvm2-lvmlockd

6. Clear stuck tasks in management node task queue

• Action: Check the ZStack Cloud UI Task Center for long-running Running or Pending tasks. Restart management node services if necessary to reset the queue.

7. Retry VM creation

• Action: Re-trigger the VM creation request in ZStack Cloud Web UI.

Verification

1. Run lvs on the storage host to confirm that the Logical Volume (LV) is in an active state.
2. Run the following command to verify that no residual locks remain:

   lvmlockd -L 2>&1 | grep lvm_

3. Re-initiate VM creation via the Web UI and confirm the task completes successfully with the VM transitioning to Running status.