Applicable Scenario

• Product: ZCF
• Component: ZStack Cloud
• ZStack Cloud versions: from 4.8.x to 5.5.x
• Operating system: h84r
• Deployment: dual management nodes
• Trigger condition: upgrade the dual-management-node platform to 5.5.x and update qemu/libvirt to LTS versions.

Symptoms

• The current platform is 4.8.x, which is not an LTS release.
• The platform must be upgraded to 5.5.x.
• The upgrade includes the control plane and qemu and libvirt components.
• Compatibility risks include experimental repo, residual userspace-rcu, and missing virtio-vga.

Diagnostic Procedure

1. Run readiness checks and inspect check.log before the upgrade.
2. After the platform upgrade, reconnect one non-critical host first for grayscale acceptance.
3. Before component upgrades, confirm whether any special compatibility scenario applies.

Cause

This is a complete dual-management-node grayscale upgrade. In addition to upgrading qemu/libvirt, it must address historical experimental repo use, package remnants, and missing migration modules. Perform the work in stages and increase scope only after validation.

Resolution

1. Preparation

• Risk level: high.
• Apply only to x86 + h84r environments.
• Disable global high availability.
• Reserve one non-critical host for grayscale reconnection validation.
• Stop all backup tasks.

2. Environment preparation
3. Run the readiness check:

   tar -zxvf zstack-upgrade-check-master.tar.gz
   cd zstack-upgrade-check-master && bash exec_tool.sh -p <management_node_admin_password>
   cat ./log/check.log

• Expected result: the upgrade environment has no blocking item.

2. Adjust database timeouts:

   wait_timeout=28800
   interactive_timeout=28800

Then run:

   systemctl restart mariadb.service

• Expected result: database timeout risk is reduced during the upgrade.

3. Update global configuration:

   UpdateGlobalConfig category=storageDevice name=enable.multipath value=ignore
   UpdateGlobalConfig category=applianceVm name=connect.timeout value=1200
   UpdateGlobalConfig name=update.os.parallelismDegree category=cluster value=5 or 10
   UpdateGlobalConfig name=update.os.parallelismDegree category=host value=10

• Expected result: upgrade and reconnection parameters are prepared.

4. Back up the dual-management-node database:

   zstack-ctl dump_mysql --file-name zstack-baskup-master-20260122
   zstack-ctl dump_mysql --file-name zstack-backup-slave-20260122

• Expected result: a platform rollback basis is available.

3. Grayscale-upgrade the platform to 5.5.x
4. Update repositories on both management nodes:

   bash zstack-upgrade -r ZStack-Cloud-x86_64-DVD-5.3.42-h84r.iso

• Expected result: upgrade repositories are switched.

2. Identify the node holding the VIP:

   zsha2 status

• Expected result: the HA-upgrade execution node is identified.

3. Extract and authorize the HA package:

   tar zxvf ZStack-Multinode-HA-Suite.tar.gz
   chmod +x zsha2 zstack-hamon

• Expected result: HA tools are ready.

4. Stop MN services on both management nodes:

   zsha2 stop-node

• Expected result: nodes enter the upgrade state.

5. On the VIP node, run:

   ./zsha2 upgrade-ha -gencfg=true

• Expected result: the HA suite is upgraded.

6. Perform the platform grayscale upgrade:

   zsha2 upgrade-mn -grayscale -yes -db-root-pw <database_root_password> -peerpass <peer_database_root_password> ZStack-Cloud-installer-5.5.0.bin

• Expected result: the platform is upgraded to 5.5.x.

7. Check after the upgrade:

   zstack-ctl status
   zsha2 status

• Expected result: management services and HA are normal.

4. Grayscale acceptance
5. Log in to the UI and verify that hosts, primary storage, image servers, and other resources are connected.

• Expected result: control-plane resources are normal.

2. Manually reconnect one non-critical host first.

• Expected result: the host upgrade status becomes Upgraded.

3. Check:

   zstack-cli QueryAgentVersion

• Expected result: the Agent version is recorded correctly.

4. Check by storage type:

• SharedBlock: vgs, lsblk, multipath -ll, and I/O error logs
• Ceph: ceph -s
• Local storage: xfs and device and path logs
• VMs: check network connectivity with console ping
• Expected result: workloads and storage are normal.

5. Reconnect the other hosts in sequence if no issue is found.
6. Disable the grayscale switch after the global upgrade:

   QueryGlobalConfig category=upgradeControl name=grayscaleUpgrade
   UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false

7. Enable global high availability.

• Expected result: the platform-upgrade phase is complete.

5. Upgrade qemu / libvirt in stages
6. Log in to zstack-cli and enable the grayscale variable first:

   QueryGlobalConfig category=upgradeControl name=grayscaleUpgrade
   UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=true

• Expected result: the component grayscale-upgrade switch is enabled.

2. Upgrade one host first:

• qemu-storage-daemon
• qemu-kvm
• libvirt
• glib2
• Expected result: validation completes on one host first.

3. Disable the grayscale variable after validation succeeds and upgrade the cluster.

• Expected result: the batch component upgrade starts.

4. Use:

   QueryLongJob uuid=<job_uuid>

Monitor upgrade progress.

5. Check versions in batches:

• Use batch SSH to run rpm -q qemu-kvm libvirt glib2.
• Expected result: host versions are consistent.

6. After the qemu upgrade, migrate or restart VMs, then run:

   virsh qemu-monitor-command $VMuuid --hmp info version

Check the actual version.

• Expected result: VMs use the new QEMU version.

6. Special compatibility items
7. If the environment was upgraded through the experimental repository, enable it again:

   UpdateGlobalConfig name="zstack.experimental.repo" value=true category=cluster

2. If the environment uses ZCE-X 5.2.x, verify that userspace-rcu and userspace-rcu-devel are fully removed.
3. If cross-cluster migration on ARM reports missing virtio-vga, manually install qemu-system-x86_64 on the new cluster and restart libvirtd.

Verification

• Verify that the platform version is 5.5.x.
• Verify that the first grayscale host and subsequently reconnected hosts are normal.
• Verify that QueryAgentVersion is correct.
• Verify that qemu/libvirt/glib2 match the target versions.
• Verify that no special compatibility conflict occurs.

Risks and Rollback

• Risk: management nodes are unavailable during the upgrade; resources and third-party APIs cannot be used.
• Risk: component upgrades involve multiple compatibility conditions; failure to identify them can affect migration, host addition, or dependencies.
• Risk level: high.
• Rollback: return dual management to single management, restore the previous repository and installer, and execute the complete restore_mysql rollback path.