Resource operations fail due to lvmlock result -28 on SharedBlock storage

Issue/Introduction

After a large amount of LV lock history accumulates in lvmlockd, the lvmlockctl output may exceed the buffer limit and report result -28. As a result, the platform may be unable to obtain complete VG lockspace information, causing resource operations that require LVM changes to fail.

Symptoms

• Volume snapshot creation fails.
• VM cloning, volume expansion, or other resource operations that involve LVM changes fail.
• Logs may contain lvmlockctl result -28, an error indicating that the VG lockspace cannot be found, or an error indicating that an LV cannot be deactivated.
• The SharedBlock primary storage remains accessible, but some resource operations cannot continue.

Environment

• Product: ZStack Cloud.
• Storage type: SharedBlock storage.
• Affected versions: Platform versions without automatic lvmlock detection and recovery optimizations, such as 5.3.x.
• Components: lvmlockd, sanlock, and the LVM lockspace on hosts.

Diagnosis

ZStack Technical Support checks the following commands on the host associated with the error:

lvmlockctl -i | head
lvmlockctl -i | wc -l
systemctl status sanlock lvmlockd
sanlock status

If lvmlockctl returns result -28 or cannot query an expected VG lockspace, use the management node and host logs to determine whether the issue matches this scenario.

Cause

The lvmlockctl dump buffer has limited capacity. When a large amount of LV lock history accumulates in lvmlockd, the query output may exceed the limit and be truncated. As a result, the platform cannot read complete VG lockspace information, which affects snapshots, expansions, clones, and other operations that change the LVM state.

Recommended actions

Temporary recovery

1. Disable global VM high availability (VM HA) in the ZStack UI and confirm that the setting has taken effect.
2. Record the current VM locations and the VG, LV, and lock states.
3. Reset the locking services on the affected host:

pkill -9 sanlock
pkill -9 lvmlockd
systemctl start sanlock
systemctl start lvmlockd

4. Restore the lockspace for each affected SharedBlock VG. Replace <VG_UUID> with the actual VG UUID of the SharedBlock primary storage:

vgchange --lock-start <VG_UUID>

If multiple VGs are affected, run the command separately with each VG UUID:

vgchange --lock-start <VG_UUID_1>
vgchange --lock-start <VG_UUID_2>

5. After completing verification, restore global VM HA according to the change plan.

Long-term solution

Upgrade the platform to a version that includes lvmlock handling optimizations. For a stable production release, upgrading to ZStack Cloud 5.4.12 is recommended.

This version enables the platform to detect related lvmlockctl buffer exceptions and automatically recover lvmlockd. During recovery, the platform does not restart sanlock and preserves the existing sanlock lock state, avoiding a manual locking-service reset for this exception. Confirm the actual target version after assessing the customer environment.

Verification

Run the following checks to confirm that the locking services and VG lockspace have recovered:

systemctl status sanlock lvmlockd
lvmlockctl -i | head
sanlock status

• Both sanlock and lvmlockd are running.
• lvmlockctl -i no longer returns result -28.
• The lockspace of each affected VG can be queried normally.
• After confirming that the lock state is normal, retry the original failed operation, such as creating a snapshot, and confirm that it succeeds.

Related information

• Recommended stable production release: ZStack Cloud 5.4.12.