Scenario

• Product: ZCF
• Component: ZStack Cloud
• Versions: 4.4.x, 4.8.x
• Storage: Local storage
• Operating system: C76
• Deployment: Dual management nodes
• Trigger: The current release is not an LTS release and must be upgraded to 4.8.x.

Symptoms

• The platform release is old and no longer meets some feature and maintenance requirements.
• ZStack Cloud, QEMU, and libvirt components must be upgraded together.
• The upgrade includes grayscale management, host reconnection, and compatibility evaluation.

Assessment

1. Run the pre-upgrade check script before the change.
2. After upgrade, verify management nodes, hosts, primary storage, and workloads.
3. Verify target qemu, libvirt, and glib2 versions.

Cause

4.4.x is not an LTS release. Upgrade to 4.8.x with the associated virtualization components to improve stability and maintainability.

Resolution

1. Preparation

• Risk level: High.
• Run zstack-upgrade-check-master, disable global high availability, and select one non-critical host for early reconnection validation.
• Upload the ISO, upgrade script, HA package, and bin files to both management nodes and verify md5 / commit.
• Set the following in /etc/my.cnf, then restart mariadb.service:

     wait_timeout=28800
     interactive_timeout=28800

• Set enable.multipath=ignore, connect.timeout=1200, update.os.parallelismDegree cluster=5/10, and update.os.parallelismDegree host=10.
• Stop all backup jobs and back up both databases:

     zstack-ctl dump_mysql --file-name 4.x-mn1
     zstack-ctl dump_mysql --file-name 4.x-mn2

2. Upgrade ZStack Cloud through grayscale deployment.
3. Update repositories on both management nodes:

      bash zstack-upgrade -r ZStack_XXXX_Cloud-x86_64-DVD-4.8.36-c76.iso

2. Locate the VIP node with zsha2 status, extract the HA package, and run:

      tar zxvf ZStack-Multinode-HA-Suite.tar.gz
      chmod +x zsha2 zstack-hamon
      zsha2 stop-node
      ./zsha2 upgrade-ha -gencfg=true

3. Upgrade management nodes:

      zsha2 upgrade-mn -grayscale -yes -db-root-pw $mysql_root_password -peerpass $peer_mysql_root_password $zstack_bin

• Expected result: Management nodes are upgraded to 4.8.x.

4. Verify services with zstack-ctl status and zsha2 status, then validate one non-critical host, kvmagent, VM networking, workloads, and zstack-cli QueryAgentVersion before reconnecting all remaining hosts.
5. Disable grayscale mode and enable global high availability:

      UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false

3. Upgrade qemu-storage-daemon, qemu-kvm, libvirt, and glib2 at cluster scope. Use QueryLongJob to check progress and SSH to verify versions. Verify each VM QEMU version with:

   virsh qemu-monitor-command $VMuuid --hmp info version

Stop backup tasks before QEMU upgrade; use pkill qemu-nbd on the backup server if a backup task remains. Restart or migrate VMs for the new QEMU version to take effect. Update zstack-virt if needed; restart no stat VMs during a business window. Evaluate ARM, experimental repositories, qemu-system-x86_64, and CPUID authorization compatibility.

Verification

• Verify that the platform is accessible and management nodes are normal.
• Verify normal host, primary-storage, and workload status.
• Verify target qemu, libvirt, and glib2 versions.
• Verify that restarted or migrated VMs use the new QEMU version.

Risks and rollback

• Risk: The platform can become inaccessible and APIs unavailable during upgrade.
• Risk: Browser cache can cause domain-access issues after the UI framework upgrade.
• Risk: QEMU/libvirt upgrades have backup-task, no stat, CPUID authorization, ARM repository, and other compatibility boundaries.
• Risk level: High.
• Rollback: Decouple dual management to single management, restore the old repository, reinstall the previous release, restore the database, and recover platform access.