Scenario

• Product: ZCF
• Component: ZStack Cloud
• Version: 5.4.x
• Operating system: H84r
• Storage: XSCALEROS_6.4.x
• Deployment: Dual management nodes
• Trigger: Local image upload is abnormal and requires a ZStack Cloud upgrade.

Symptoms

• Local image upload is abnormal in the Cloud 5.4.x environment.
• A platform grayscale upgrade is required.
• QEMU and libvirt are already current; this change does not upgrade components, operating system, or kernel.

Assessment

1. Run the precheck script to confirm health before upgrade.
2. Verify management-node state, resource status, and workloads after upgrade.
3. Validate one manually reconnected non-critical host before reconnecting all hosts.

Cause

Local image upload is abnormal in Cloud 5.4.x, so upgrade the platform to 5.4.7.

Resolution

1. Preparation

• Risk level: High.
• Run zstack-upgrade-check-master.
• Disable global high availability.
• Select one non-critical host for early reconnection validation.
• Upload the ISO, upgrade script, HA components, and bin files to both management nodes and verify md5 and commit.
• Adjust:
• enable.multipath=ignore
• connect.timeout=1200
• update.os.parallelismDegree cluster=5/10
• update.os.parallelismDegree host=10
• Stop all platform backup jobs.
• Back up both databases:

~~~bash zstack-ctl dump_mysql --file-name 4.x-mn1 zstack-ctl dump_mysql --file-name 4.x-mn2 ~~~

2. Grayscale upgrade
3. Update the repository on both management nodes:

~~~bash bash zstack-upgrade -r ZStack-Cloud-x86_64-DVD-5.4.7-h84r.iso ~~~

• Expected result: The repository is updated.

2. Locate the VIP node:

~~~bash zsha2 status ~~~

• Expected result: The VIP management node is identified.

3. Extract the HA package and grant execution permission:

~~~bash tar zxvf Multinode-HA-Suite-zsha2_5.4.0.tar.gz chmod +x zsha2 zstack-hamon ~~~

• Expected result: HA upgrade materials are ready.

4. Stop dual-management services:

~~~bash zsha2 stop-node ~~~

• Expected result: The platform is ready for upgrade.

5. Upgrade HA on the VIP node:

~~~bash ./zsha2 upgrade-ha -gencfg=true ~~~

• Expected result: The HA suite is upgraded.

6. Upgrade ZStack Cloud:

~~~bash zsha2 upgrade-mn -grayscale -yes -db-root-pw $mysql_root_password -peerpass $peer_mysql_root_password ZStack-Cloud-installer-5.4.7.bin ~~~

• Expected result: Management nodes are upgraded to 5.4.7.

7. Check status:

~~~bash zstack-ctl status zsha2 status ~~~

• Expected result: Management-node status is normal.

8. Check resource status in the UI.
9. Reconnect one non-critical host manually, verify its VM workloads, then reconnect the remaining hosts one by one.
10. Check agent version:

~~~bash zstack-cli QueryAgentVersion ~~~

11. Check XSKY storage health:

~~~bash ceph -s ~~~

12. Verify VM networking and workloads.
13. Disable grayscale mode:

~~~text UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false ~~~

14. Enable global high availability.

• Expected result: Upgrade is complete and normal operation is restored.

Verification

• Verify normal management-node, host, primary-storage, and image-server state.
• Verify the non-critical host reconnection.
• Verify XSKY storage health.
• Verify that local image upload is restored.

Risks and rollback

• Risk: Management nodes, new resource provisioning, and third-party APIs are unavailable during upgrade.
• Risk: Obsolete browser UI assets can cause domain-access issues until browser cache is cleared.
• Risk level: High.
• Rollback: Decouple dual management, restore the old repository and release, restore databases, and then restore platform access.