Scenario

• Product: ZCF
• Component: ZStack Cloud
• ZStack Cloud version: 5.4.x
• Operating system: H84r
• Storage involved: ZCE-X
• Deployment: Dual management nodes
• Trigger: The dual-management platform must be upgraded through grayscale deployment to a newer LTS patch release.

Symptoms

• The current platform version is 5.4.x.
• The target version is 5.4.x.
• This change upgrades only ZStack Cloud, not qemu/libvirt/kernel.
• One host must be reconnected first to validate the agent and workload state.

Assessment

1. Run the precheck and inspect check.log before upgrade.
2. Reconnect one non-critical host for grayscale validation after upgrade.
3. Reconnect remaining hosts only after the first validation passes.

Cause

Although this change does not modify core virtualization-component versions, it is still a dual-management grayscale upgrade. The key risks are agent upgrade, management-node switching, database timeouts, and environmental compatibility. Complete precheck, preparation, and grayscale validation are required.

Resolution

1. Preparation

• Risk level: High.
• Applicable only to x86 + H84r environments.
• Disable global high availability.
• Select one non-critical host for grayscale reconnection validation.
• Stop all backup jobs.

2. Precheck and environment preparation
3. Run the precheck package:

      tar zxvf zstack-upgrade-check-master.tar.gz
      cd zstack-upgrade-check-master && bash exec_tool.sh -p [MN_admin_password]
      cat ./log/check.log

• Expected result: Compatibility and blocking conditions are checked.

2. Upload the ISO, upgrade script, HA package, and bin file, and verify md5/commit.

• Expected result: Upgrade materials are correct.

3. Set database timeouts, then restart MariaDB:

      wait_timeout=28800
      interactive_timeout=28800

      systemctl restart mariadb.service

• Expected result: Database timeouts are avoided during upgrade.

4. Adjust global configuration for the environment:

      UpdateGlobalConfig category=storageDevice name=enable.multipath value=ignore
      UpdateGlobalConfig category=applianceVm name=connect.timeout value=1200
      UpdateGlobalConfig name=update.os.parallelismDegree category=cluster value=5 or 10
      UpdateGlobalConfig name=update.os.parallelismDegree category=host value=10

• Expected result: Parameters are prepared for grayscale upgrade and host reconnection.

5. Back up the databases on both management nodes.

• Expected result: A platform rollback baseline is available.

3. Perform grayscale upgrade
4. Update the repository on both management nodes:

      bash zstack-upgrade -r ZStack_Alibaba_Cloud-x86_64-DVD-5.4.6-h84r.iso

• Expected result: The upgrade repository is switched.

2. Locate the VIP node:

      zsha2 status

• Expected result: The node for upgrade-ha is identified.

3. Extract the HA package and grant execute permission:

      tar zxvf ZStack-Multinode-HA-Suite.tar.gz
      chmod +x zsha2 zstack-hamon

• Expected result: HA upgrade tools are executable.

4. Stop management-node services:

      zsha2 stop-node

• Expected result: Management nodes are ready for upgrade.

5. Upgrade the HA suite on the VIP node:

      ./zsha2 upgrade-ha -gencfg=true

• Expected result: The HA suite is upgraded.

6. Perform grayscale upgrade:

      zsha2 upgrade-mn -grayscale -yes -db-root-pw $mysql_root_password -peerpass $peer_mysql_root_password $zstack_bin

In a single-management environment, bash zstack-installer.bin -u --grayscale true is for reference.

• Expected result: ZStack Cloud is upgraded to 5.4.x in grayscale mode.

4. Grayscale validation after upgrade
5. Check status:

      zstack-ctl status
      zsha2 status

• Expected result: Management-node and HA status are normal.

2. In the UI, verify that hosts, primary storage, image servers, and other resources are connected.

• Expected result: Platform resources are connected.

3. Reconnect one non-critical host manually.

• Expected result: Its agent is upgraded without affecting workloads.

4. Check the agent-version record:

      zstack-cli QueryAgentVersion

• Expected result: The version record is correct.

5. Check the relevant storage type:

• SharedBlock: vgs, multipath, SAN links, and /var/log/messages.
• Ceph: ceph -s.
• Local storage: xfs, device names, and mount-path logs.
• Expected result: Host storage and workloads are normal.

6. Reconnect remaining hosts after grayscale validation passes.
7. Disable grayscale mode:

      QueryGlobalConfig category=upgradeControl name=grayscaleUpgrade
      UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false

• Expected result: Grayscale mode is closed.

8. Enable global high availability.

• Expected result: The upgrade is complete.

Verification

• Verify normal zstack-ctl status and zsha2 status.
• Verify normal workloads on the first reconnected host.
• Verify the QueryAgentVersion record.
• Verify that all platform resources are connected.

Risks and rollback

• Risk: Management nodes, APIs, and new resource provisioning are unavailable during upgrade.
• Risk: Inadequate prechecks can cause host or primary-storage connection failures during grayscale deployment.
• Risk: Browser cache can prevent the upgraded UI from loading until the cache is cleared.
• Risk level: High.
• Rollback includes decoupling dual management to single management, stopping zsha2 / keepalived / zstack-ha, running stop slave / reset slave all, rewriting DB.url, removing /usr/local/zstack, and restoring the previous repository, installer, and database with restore_mysql.