Applicable scenario

• Product: ZCF
• Component: Cloud
• Versions: 4.3.x, 3.8.x
• Deployment topology: dual management nodes
• Trigger: Upgrade sanlock to address stalled vgchange and locks not automatically released after a storage failure.

Symptoms

• vgchange stalls.
• Locks are not automatically released after a storage failure.
• The underlying sanlock component and associated lock service require upgrade.

Determination method

1. Confirm that physical-host and storage ping intervals have been extended before upgrade.
2. After clearing lock processes, check whether sanlock and lvmlockd processes have exited.
3. After upgrade, check that sanlock is version 3.8.x and vgs returns normally.

Cause

The purpose of this change is to address stalled vgchange and unreleased locks in a SharedBlock scenario. Upgrade the underlying sanlock component one physical host at a time, strictly controlling lock-process state and automatic reconnection before and after the upgrade.

Resolution

1. Preparation

• Risk level: high.
• Disable global VM HA.
• Set physical-host and storage ping intervals to 24h (86400s).
• Upgrade one physical host at a time; do not process multiple physical hosts concurrently.

2. Disable global VM HA.

• Expected result: HA is not triggered unexpectedly during maintenance.

3. Set physical-host and storage ping intervals to 86400 seconds in global configuration.

• Expected result: Automatic host reconnection does not overwrite the upgrade operation.

4. Force-stop lock processes and verify:

   pkill -9 sanlock && pkill -9 lvmlockd
   ps -ef | grep -E "sanlock|lvmlockd"

• Expected result: Existing lock processes are cleared and the environment can enter the upgrade state.

5. Upload the sanlock RPM package to the target physical host.

• Expected result: The upgrade package is ready.

6. Upgrade sanlock on one physical host:

   yum install ./sanlock*.rpm

• Expected result: The current node installs the new component version.

7. Check the version:

   rpm -qa | grep sanlock

• Expected result: The version is 3.8.x.

8. Start and check services:

   systemctl start sanlock.service
   systemctl start lvm2-lvmlockd
   systemctl status sanlock.service
   systemctl status lvm2-lvmlockd

• Expected result: Lock services recover normally.

9. Reconnect the physical host and check:

   vgs

• Expected result: vgs returns normally and no longer stalls.

10. After confirming one host, upgrade the remaining physical hosts in sequence.
11. Restore the ping interval configuration after all upgrades complete.
12. Perform a full check:

• vgs
• pvs
• Process check:

     ps -ef | grep -E 'lvchange|lvcreate|lvrename|lvresize|lvextend|lvremove|lvs|vgs|pvs|vgchange'

• Service check:

     systemctl status sanlock.service
     systemctl status lvm2-lvmlockd

• Expected result: All physical hosts, lock services, and LVM-related processes recover normally.

13. Enable global VM HA after confirming all resources are normal.

• Expected result: The upgrade process is complete.

Verification

• sanlock is version 3.8.x.
• vgs and pvs return normally.
• sanlock and lvm2-lvmlockd service states are normal.
• Hosts reconnect and service resources are normal.

Risks and rollback

• Risk: Upgrading the underlying lock component sanlock directly affects SharedBlock service continuity.
• Risk: If HA is not disabled or the ping interval is not extended before upgrade, automatic reconnection can interrupt the change.
• Risk level: high.
• Rollback:

1. Disable HA.
2. Clear lock processes again.
3. Run:

     yum downgrade sanlock-3.8.3-1.el7.x86_64 sanlock-lib-3.8.3-1.el7.x86_64

4. Restart sanlock/lvm2-lvmlockd and check them.
5. Reconnect the physical host.
6. Re-enable HA after resources are normal.