Applicable Scenario

• Product: ZCF
• Component: ZStack Cloud
• Versions: 5.3.x and 5.5.x
• Operating system/repository types: h84r / ky10sp3
• Deployment: dual management nodes
• Trigger condition: upgrade the platform to 5.5.x for supported sriov features and bring qemu/libvirt to the new LTS versions.

Symptoms

• The current platform version is 5.3.x.
• The target version is 5.5.x.
• qemu/libvirt versions differ between the test and production areas and must be standardized.
• The upgrade involves both h84r and ky10sp3 repositories.

Diagnostic Procedure

1. Complete readiness checks and inspect check.log before the upgrade.
2. Reconnect one non-critical host first after the platform upgrade for grayscale acceptance.
3. Check qemu/libvirt/glib2 versions and the actual VM QEMU version after component upgrades.

Cause

In addition to upgrading the control plane, this procedure aligns qemu/libvirt versions for feature support and version convergence. Dual management and dual repositories require grayscale platform upgrade and host-reconnection acceptance before the component upgrade.

Resolution

1. Preparation

• Risk level: high.
• Run the readiness-check script and check zsha2 status.
• Disable global high availability.
• Reserve one non-critical host for reconnection acceptance.
• Stop all backup tasks.

2. Environment preparation
3. Run the readiness check:

   tar zxvf zstack-upgrade-check-master.tar.gz
   cd zstack-upgrade-check-master
   bash exec_tool.sh -p [MN_admin_password]
   cat ./log/check.log

• Expected result: the upgrade environment has no blocking item.

2. Adjust database timeouts and restart mariadb:

   wait_timeout=28800
   interactive_timeout=28800

Then run:

   systemctl restart mariadb.service

• Expected result: database timeout risk is reduced during the upgrade.

3. Update global configuration:

   UpdateGlobalConfig category=storageDevice name=enable.multipath value=ignore
   UpdateGlobalConfig category=applianceVm name=connect.timeout value=1200
   UpdateGlobalConfig name=update.os.parallelismDegree category=cluster value=5 or 10
   UpdateGlobalConfig name=update.os.parallelismDegree category=host value=10

• Expected result: upgrade and host-reconnection parameters are prepared.

4. Back up both management-node databases:

   zstack-ctl dump_mysql --file-name 4.x-mn1
   zstack-ctl dump_mysql --file-name 4.x-mn2

• Expected result: a platform rollback basis is available.

3. Grayscale-upgrade the platform to 5.5.x
4. Update both repositories on the dual management nodes:

   bash zstack-upgrade -r ZStack-x86_64-DVD-5.5.1-260112-h84r-Enterprise.iso
   bash zstack-upgrade -r ZStack-x86_64-DVD-5.5.1-260112-ky10sp3-Enterprise.iso

• Expected result: management nodes have the target-version repositories.

2. Identify the VIP node:

   zsha2 status

• Expected result: the HA-upgrade execution node is identified.

3. Extract and authorize the HA package:

   tar zxvf ZStack-Multinode-HA-Suite.tar.gz
   chmod +x zsha2 zstack-hamon

• Expected result: HA upgrade tools are ready.

4. Stop MN services and upgrade the HA suite:

   zsha2 stop-node
   ./zsha2 upgrade-ha -gencfg=true

• Expected result: the dual-management HA suite is upgraded.

5. Perform the platform grayscale upgrade:

   zsha2 upgrade-mn -grayscale -yes -db-root-pw $mysql_root_password -peerpass $peer_mysql_root_password $zstack_bin

• Expected result: the platform is upgraded to 5.5.x.

6. Check after the upgrade:

   zstack-ctl status
   zsha2 status

• Expected result: management nodes and HA are normal.

4. Grayscale acceptance
5. Log in to the UI and check hosts, primary storage, image servers, and other resources.

• Expected result: all resources are connected.

2. Manually reconnect one non-critical host first.

• Expected result: the host upgrade status becomes Upgraded.

3. Check:

   zstack-cli QueryAgentVersion

• Expected result: the Agent version is recorded correctly.

4. Check by storage type:

• SharedBlock: vgs, lsblk, multipath -ll, and I/O error logs
• Ceph: ceph -s
• Management-node logs: management-server.log
• VMs: console ping connectivity
• Expected result: hosts, storage, and workloads are normal.

5. Reconnect other hosts in sequence if no issue is found.
6. Disable the grayscale variable and enable global high availability:

   QueryGlobalConfig category=upgradeControl name=grayscaleUpgrade
   UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false

• Expected result: the platform-upgrade process is complete.

5. Upgrade qemu / libvirt / glib2
6. Keep using both 5.5.x repositories on the dual management nodes.
7. Upgrade one host first:

• qemu-storage-daemon
• qemu-kvm
• libvirt
• glib2
• Expected result: validation completes on one host first.

3. After one-host validation, perform the same upgrade for the cluster.
4. Use:

   QueryLongJob uuid=xxx

Track progress.

5. Use batch SSH with rpm -q qemu-kvm libvirt glib2 to verify version consistency.

• Expected result: both test and production areas are aligned to the target versions.

6. After the qemu upgrade, migrate or restart VMs and run:

   virsh qemu-monitor-command $VMuuid --hmp info version

Check the actual QEMU version.

• Expected result: VMs run with the new QEMU version.

Verification

• Verify that the platform version is 5.5.x.
• Verify that the grayscale host and subsequently reconnected hosts are normal.
• Verify that QueryAgentVersion is correct.
• Verify that qemu/libvirt/glib2 match target versions.
• Verify that the VM QEMU version is updated.

Risks and Rollback

• Risk: management nodes are unavailable during the upgrade and cannot provision VMs or volumes.
• Risk: qemu/libvirt upgrades affect virtualization compatibility and require adequate validation.
• Risk: a qemu upgrade requires migration or restart before VMs use the new version.
• Risk level: high.
• Rollback: return dual management to single management, then restore the previous repository, installer, and database.