Applicable scenario

• Product: ZCF
• Component: Cloud
• Versions: 4.4.x, 3.8.x
• Deployment topology: dual management nodes
• Trigger: ShareBlock primary storage cannot connect, locks cannot be released on many physical hosts, vgchange stalls, and a historical sanlock defect is confirmed.

Symptoms

• ShareBlock primary storage reports a connection failure.
• Locks cannot be released normally on many physical hosts.
• The vgchange command stalls.
• The root cause is a historical known defect in sanlock.

Determination method

1. Confirm that the primary-storage connection failure is accompanied by an unreleased-lock exception.
2. Check whether sanlock/lvmlockd have unresponsive processes.
3. After upgrade, check that sanlock reaches version 3.8.x.

Cause

This is not merely connection jitter. A historical sanlock defect prevents many locks from being released, which further causes vgchange to stall and ShareBlock primary storage to fail to connect. Upgrade the underlying sanlock component.

Resolution

1. Preparation

• Risk level: high.
• Disable global VM HA.
• Back up the databases on both management nodes.
• Extend physical-host and storage ping intervals to 24h (86400s) to prevent automatic reconnection from overwriting the upgrade operation.
• Validate one host first, then roll out in batches.

2. Back up the database:

   zstack-ctl dump_mysql

• Run on both the VIP management node and the secondary node.
• Expected result: A platform-side rollback baseline is available.

3. Set physical-host and storage ping time to 86400 seconds in global configuration.

• Expected result: Host reconnection does not interfere during upgrade.

4. Force-stop unresponsive lock processes:

   pkill -9 sanlock && pkill -9 lvmlockd
   ps -ef | grep -E "sanlock|lvmlockd"

• Expected result: Stalled lock processes are cleared to allow the upgrade to continue.

5. Upgrade sanlock on one host:

   yum install ./sanlock*.rpm

• Expected result: The target host completes component replacement first.

6. Check the version:

   rpm -qa | grep sanlock

• Expected result: The version is 3.8.x.

7. After confirming the one-host procedure is normal, upgrade other hosts in batches.

• Expected result: Uncontrolled risk from broad rollout is avoided.

8. Reconnect physical hosts after upgrade.
9. Restore global HA and the original ping interval configuration.

• Expected result: Normal operating policy returns and ShareBlock reconnects stably.

Verification

• rpm -qa | grep sanlock shows 3.8.x.
• ShareBlock primary storage reconnects.
• vgchange no longer stalls.
• Hosts reconnect and services run normally.

Risks and rollback

• Risk: This is an underlying component upgrade; incorrect operations can expand the storage impact scope.
• Risk: Directly reconnecting a host in Connecting, Disconnected, or Unknown state while its path is abnormal can worsen the primary-storage problem.
• Risk level: high.
• Risk controls:
• Locate the abnormal host by UUID in the UI.
• Check the host's FC/iSCSI path, multipathing, and disk recognition for ShareBlock.
• If only one host is abnormal, isolate or maintain that host first to avoid affecting the entire primary-storage reconnection.
• Operate storage reconnection only after the host recovers.
• Rollback: No detailed RPM downgrade command is available. Prioritize one-host validation and isolation of abnormal hosts to control risk.