Scenario

• Product: ZCF
• Component: ZStack Cloud
• Version: 4.3.x
• Trigger: A thin-provisioned VM pauses during automatic expansion, together with primary-storage disconnection and blocked vgs.

Symptoms

• A VM pauses automatically and cannot start normally after stopping.
• Primary storage is disconnected.
• vgs blocks on multiple or a single node.
• The affected node has residual vgchange processes.

Assessment

1. Check whether primary storage is disconnected.
2. Check whether vgs blocks on all nodes or only the affected node.
3. Check for residual processes:

~~~bash ps -ef | grep vgchange ~~~

4. Check sanlock and lvm2-lvmlockd status and historical aio timeout and IO error/timeout events.

Cause

The VM pause is not a normal guest failure. When a thin-provisioned VM reaches its threshold, automatic lvextend can fail because of a VGLK lock exception. The volume becomes full and the VM pauses automatically. Historical sanlock aio timeout, extensive IO error/timeout, and a residual vgchange process holding flock can further aggravate lock contention.

Resolution

1. Emergency recovery
2. When a VM pauses and primary storage disconnects, check vgs on all nodes.

• Expected result: Determine whether the lock hang is cluster-wide or node-specific.

2. If vgs blocks broadly, restart lock services to release abnormal locks:

• sanlock
• lvm2-lvmlockd
• Expected result: Primary storage reconnects and vgs returns normally.

3. If the issue persists on a single node, for example only node 32.1.x blocks:

~~~bash ps -ef | grep vgchange ~~~ Locate residual vgchange processes.

• Expected result: The process holding the lock is identified.

4. Remove residual vgchange processes.

• Expected result: flock is released and subsequent lvchange/lvextend operations no longer remain blocked.

5. If the VM cannot recover on its original node, stop it and start it on another host.

• Expected result: Service is recovered first.

2. Environment health check
3. Check whether vgs and pvs return normally.
4. Check that no related residual process remains:

~~~bash ps -ef | grep -E 'lvchange|lvcreate|lvrename|lvresize|lvextend|lvremove|lvs|vgs|pvs|vgchange' ~~~

3. Check service status:

~~~bash systemctl status sanlock systemctl status lvm2-lvmlockd ~~~

• Expected result: LVM and lock services return to normal.

Verification

• Verify that primary storage reconnects.
• Verify normal vgs and pvs output.
• Verify that the paused VM can start on a healthy host.
• Verify that no abnormal vgchange or related residual process remains.

Risks and follow-up actions

• Risk: If residual vgchange processes are not removed, the issue can recur after service restart.
• Risk: When a thin LV is full, paused workloads directly affect database-type workloads.
• Recommended long-term actions:

1. Recommended long-term remediation: upgrade sanlock to address locks not released automatically and blocked vgchange.
2. Upgrade ZStack Cloud to the latest 4.8.x.x LTS release to improve sharedblock lock handling.
3. Continue environment health checks.