Applicable Scenario

• Product: ZCF
• Component: ZStack Cloud
• ZStack Cloud version: 4.8.x
• Operating system/repository environment: H84R
• Trigger condition: known issues in the current platform require an upgrade to 4.8.x by using the one-click upgrade script.

Symptoms

• The current platform version is 4.8.x.
• The upgrade target is 4.8.x.
• The source mentions 4.8.x in some locations, while the upgrade script and ISO point to 4.8.x.
• After the upgrade, the following also require handling:
• qemu/libvirt components
• Host management reconnection
• Dedicated load-balancer reconnection
• Supporting upgrades for ZBS primary storage

Diagnostic Procedure

1. Use the readiness-check package to confirm environment health.
2. After the one-click upgrade completes, check host, primary-storage, image-server, and high-availability status.
3. Complete component upgrades and dedicated load-balancer reconnection verification.

Cause

The one-click script upgrades the control plane to 4.8.x to address known issues and improve maintainability. It does not complete virtualization-component upgrades, host reconnection, or storage and load-balancer follow-up work, which must be performed separately.

Resolution

1. Pre-upgrade environment checks
2. Check environment health with the upgrade readiness package:

   tar zxvf zstack-upgrade-check-master.tar.gz
   cd zstack-upgrade-check-master
   bash exec_tool.sh -p [MN_admin_password]
   cat ./log/check.log

• Expected result: environment health and check.log are confirmed.

2. Check the following:

• Cloud Router / VPC
• AD/LDAP
• SharedBlock
• Ceph
• VM split-brain checks
• HTTPS
• SSH upgrade method
• Historical API audit information
• Expected result: the environment meets upgrade prerequisites.

3. Perform an additional check:

   zsha2 status

• Check VM instance status, resource status, and platform high availability.
• Expected result: the platform operates stably.

2. Preparation before the change
3. Disable global high availability.
4. Back up the database on both management nodes.

• Expected result: a rollback basis is available.

3. Download and verify the one-click upgrade script:

• cloud-upgrade-scripts_0.2.x.tar.gz / cloud-upgrade-scripts_0.3.x.tar.gz
• Reference md5:

     f8965606d47babad93384b0be8cf4a55

• Expected result: the script package passes verification.

4. Download and verify the upgrade ISO:

   ZStack_Alibaba_Cloud-x86_64-DVD-4.8.36-h84r.iso
   MD5: fcad1a7ccf65282bdec516c4b6010d84

• Expected result: the installation media is complete and usable.

3. Upgrade ZStack Cloud with the one-click script
4. Upload the upgrade script and ISO file to both MN nodes.
5. Extract the upgrade script:

   mkdir -p /root/upgrade && tar -xzvf /root/cloud-upgrade-scripts_v0.3.tar.gz -C /root/upgrade

• Expected result: the upgrade-script directory is extracted.

3. Copy the ISO to the script directory:

   cp ZStack_Alibaba_Cloud-x86_64-DVD-4.8.36-h84r.iso upgrade/cloud-upgrade-scripts/

4. Run the one-click upgrade:

   bash upgrade/cloud-upgrade-scripts/cloud-upgrade.sh ZStack_Alibaba_Cloud-x86_64-DVD-4.8.36-h84r.iso

• Select the required options when prompted.
• Enter the admin password and required configuration.
• Expected result: the control plane is upgraded to 4.8.x.

4. Complete component and environment follow-up work
5. Use the preceding procedure as a reference to upgrade qemu/libvirt components.
6. After the upgrade, check:

• Host management connectivity
• VM resource status
• Platform service status
• Expected result: fundamental platform functions are restored.

3. Additional notes:

• Host management may need manual reconnection after the upgrade.
• Dedicated load balancers require reconnection to take effect.
• When ZBS is used as primary storage, upgrade ZBS as well.
• Expected result: all external dependencies are completed.

5. Component risk-control requirements
6. When subsequently upgrading QEMU and Libvirt, strictly follow these requirements:

• Stop all backup tasks.
• If backup processes are running, run the following on all backup servers:

     pkill qemu-nbd

• After a QEMU upgrade, migrate or restart VMs for the new version to take effect.
• After a Libvirt upgrade, check VMs in no state.
• Non-ZCE-X environments can use the online hot-upgrade tool.
• After the platform upgrade, run:

     zstack-cli UpdateClusterOS uuid=<cluster_uuid> updatePackages=zstack-virt

• Expected result: component-upgrade risk is controlled.

Verification

• Verify that the one-click upgrade script completes and the platform version is 4.8.x.
• Verify that hosts, primary storage, image servers, and high availability are normal.
• Verify that hosts requiring manual reconnection are connected again.
• Verify that dedicated load balancers work after reconnection.
• Verify that ZBS primary-storage environments complete supporting upgrades.

Risks and Notes

• Risk: management nodes are unavailable during the upgrade and cannot provision new resources.
• Risk: after the one-click upgrade, host connections and supporting components can still require manual handling.
• Risk: QEMU cannot be directly rolled back.
• Risk level: high.

Rollback

The rollback path returns dual management to single management. Key steps are:

1. Prepare the pre-upgrade ISO, installer, and dual-management HA suite on both management nodes.
2. Back up the database again.
3. Use zsha2 status to identify the VIP node and read chrony.serverIp.
4. Stop and disable dual management:

   zsha2 stop-node
   systemctl disable keepalived
   systemctl disable zstack-ha

5. Run in the database:

   stop slave;
   reset slave all;
   show slave status;

6. Point DB.url back to the real single-management-node IP.
7. Remove the current environment and restore the original repo, installer, and database.
8. Reinstall the original zsha2 suite on the VIP node.
9. Log in to VIP:5000 and check platform resources and workload status.