Applicable scenario

• Product: ZCF
• Components: ZCE-X, Cloud
• ZCE-X version: 5.2.x
• Connected ZStack Cloud version: ZStack Cloud 4.4.x
• ZStackCube version: Upgrade ZStackCube 2.2.x to 2.4.x
• Current versions:
• ZStackCube 2.2.x
• ZCE-X 5.2.x
• Cloud 4.4.x
• qemu 4.2.x
• libvirt 4.9.x
• Target versions:
• ZStackCube 2.4.x
• ZCE-X 5.2.x
• qemu 6.2.x
• libvirt 8.0.x
• Operating system: C76
• Deployment topology: 1 cluster, 10 physical hosts, 1 primary storage
• Trigger: File storage writes are slow, and a transition upgrade to 5.2.x is required before a future V6 storage upgrade.

Symptoms

• The environment uses old versions.
• File storage writes are slow.
• The platform, virtualization components, and storage require coordinated upgrades.

Determination method

1. Before upgrade, confirm platform, Ceph, etcd, ES, Postgres synchronization, host resources, and passwordless access are normal.
2. After platform upgrade, check management nodes, hosts, primary storage, image servers, and VMs.
3. After storage upgrade, check ceph -s, xms-cli -v, xdc, xmsd, docker ps, and UI modules.

Cause

Slow file storage is one cause, and the storage side is recommended for future V6 upgrade. The current version cannot upgrade directly to V6, so first upgrade platform and storage to supported transition versions.

Resolution

1. Preparation

• Risk level: high.
• Put platform resources in maintenance mode and disable global VM HA.
• Run pre_upgrade_self_check before platform upgrade.
• Stop all backup tasks.
• Back up dual-management databases and the zstack-upgrade script.
• Before qemu upgrade, clear possible running qemu-nbd on the backup server.

2. Grayscale-upgrade ZStackCube to 2.4.x
3. Run pre_upgrade_self_check.

• Expected result: Cross-version upgrade risks are identified.

2. Disable global HA and choose one non-critical physical host for first reconnection validation.
3. Upload ISO, upgrade package, HA components, and bin files to both management nodes, then verify md5 and zsha2 commit.
4. Modify /etc/my.cnf:

   wait_timeout=28800
   interactive_timeout=28800

Then restart mariadb.service.

5. Back up databases on both management nodes:

   zstack-ctl dump_mysql --file-name mn1
   zstack-ctl dump_mysql --file-name mn2
   cp /usr/local/bin/zstack-upgrade /root/zstack-upgrade-bk

6. Update the repository:

   bash zstack-upgrade -r Cube-x86_64-DVD-2.4.17-c76-Enterprise-upgrade-241024.iso

7. Query the VIP node:

   zsha2 status

8. Extract the HA package and grant permissions:

   tar zxvf ZStack-Cube-Multinode-HA-Suite-2.4.17.tar.gz
   chmod +x zsha2 zstack-hamon

9. Stop MN services and upgrade HA:

   zsha2 stop-node
   ./zsha2 upgrade-ha -gencfg=true

10. Grayscale-upgrade ZStackCube:

    zsha2 upgrade-mn -grayscale -yes xx.bin

11. Check after upgrade:

    zstack-ctl status
    zsha2 status

12. Log in to the UI, check resource state, and reconnect the non-critical host first.
13. Check agent versions:

    zstack-cli QueryAgentVersion

14. On Ceph storage, check:

• ceph -s
• Firewall state
• Physical-host-to-Ceph network connectivity
• Heartbeat-volume access

15. Reconnect remaining physical hosts in sequence after no exception.
16. Disable grayscale variables and enable global HA.

• Expected result: Platform upgrade completes.

3. Upgrade qemu/libvirt/glib2 components
4. Stop all backup tasks. If backup processes are running, run on the backup server:

   pkill qemu-nbd

2. Upgrade one physical host first:

• qemu-storage-daemon
• qemu-kvm
• glib2
• libvirt

3. Upgrade remaining physical hosts only after single-host validation succeeds.
4. Query progress with QueryLongJob.
5. Restart or migrate VMs after upgrade to load the new qemu.
6. Check VM qemu version:

   virsh qemu-monitor-command $VMuuid --hmp info version

• Expected result: Virtualization component upgrade completes.

4. Offline-upgrade ZCE-X to 5.2.x
5. Before upgrade, confirm:

• ceph -s is healthy
• etcd is healthy
• ES state is green
• Passwordless SSH works
• System-volume space is sufficient
• /opt/sds permissions are 755
• umask=0022
• Postgres synchronization is normal
• Historical alerts are cleared
• Cluster version is confirmed

2. Use xms-manage db list to locate the primary database node.
3. Download and extract the upgrade package on the primary database node.
4. Run:

   ./upgrade.sh

5. To skip Ceph status checks:

   ./upgrade.sh --nc

6. If the upgrade is interrupted:

   ./upgrade.sh --continue

7. If the Prometheus port conflicts, change Prometheus.exporterPort in zstack.properties and restart MN services.

• Expected result: Storage upgrades to 5.2.x.

5. Post-upgrade checks and completion
6. Check:

   ceph -s
   xms-cli -v
   systemctl status xdc
   systemctl status xmsd
   docker ps

Check UI modules including block storage, file storage, Resource Management, and Monitoring Management.

2. Re-enable physical hosts and VMs in the Ceph cluster, then enable global HA.
3. Run the inspection script for environment health.

• Expected result: The platform, virtualization components, and storage recover health after upgrade.

Verification

• Management services, Ceph cluster, and storage pools are healthy.
• Platform resources and service VMs are normal.
• xms-cli -v, qemu, and libvirt reach target versions.
• The inspection result contains no critical alerts.

Risks and rollback

• Risk: Management nodes are unavailable during platform upgrade, and component upgrade has significant compatibility impact.
• Risk: VMs must migrate or restart after qemu/libvirt upgrade, and the qemu version cannot be rolled back.
• Risk: Storage upgrades are not normally rolled back; if interrupted, analyze the cause and continue with ./upgrade.sh --continue.
• Risk level: high.
• Rollback: A complete dual-management rollback plan is available only for ZStackCube; rolling back storage and qemu is not recommended without careful confirmation.