Environment

• Product: HCI
• Component: Ultimate
• Objects: Dual management nodes, QEMU, libvirt, and management node high availability
• Current version: 2.4.17-c76 (Cloud-4.8.15)
• Target version: 2.4.26-c76 (Cloud-4.8.36)
• Trigger: The platform version is outdated. The cloud platform and virtualization components must be upgraded while retaining a rollback path in case of failure.

Symptoms

• The current HCI Ultimate version is 2.4.17 and is outdated.
• The platform must be upgraded to 2.4.26, together with the QEMU and libvirt components.
• If the upgrade fails, rollback to the original version must be supported.

Diagnosis

1. Confirm that the environment is within the scope of this article, and record the current and target versions, system architecture, operating system, deployment model, storage type, and other key information.
2. Use the pre-upgrade check package to check the health of the environment.
3. Confirm that VM instances, resources, and high availability are normal.
4. After the upgrade, check the canary upgrade switch, physical resources, VPC connections, services, and VM instance status.

Cause

The objective is to upgrade the platform version. To upgrade the platform and its virtualization components, upgrade HCI Ultimate and QEMU/libvirt, and prepare a rollback plan for dual-management-node decoupling and database recovery.

Resolution

1. Prerequisites

• Risk level: Medium
• Minimize the impact on existing VM services during the change.
• Before the upgrade, complete the environment check and back up the database on both management nodes.

2. Pre-upgrade environment check

1. Download and extract the check package.

   tar xzf zstack-upgrade-check-*.tar.gz
   cd zstack-upgrade-check-*
   bash upgrade_precheck.sh -k zstack2026 [target-version]

• For example: bash upgrade_precheck.sh -k zstack2026 5.4.7
• Expected result: Risk notices and precautions are displayed, and a precheck log is generated.

2. Check the cloud platform high availability status.

   zsha2 status

• Expected result: High availability is normal.

3. Cloud platform upgrade procedure

1. In the management node UI, disable global high availability for VM instances, and back up the database on both management nodes.

   zstack-ctl dump_mysql --file-name backup-master
   zstack-ctl dump_mysql --file-name backup-slave

• Expected result: The database backups are complete and high availability is disabled.

2. Prepare the upgrade script and ISO, and extract the upgrade script on the management node.

   mkdir -p /root/upgrade && tar -xzvf /root/cloud-upgrade-scripts_v0.4.tar.gz -C /root/upgrade
   cp Cube-x86_64-DVD-2.4.26-260206-c76-Enterprise.iso upgrade/cloud-upgrade-scripts/
   bash upgrade/cloud-upgrade-scripts/cloud-upgrade.sh Cube-x86_64-DVD-2.4.26-260206-c76-Enterprise.iso

• Follow the prompts to select the operations, enter the admin password, and select the related configuration.
• Expected result: The platform upgrade procedure is complete.

3. Upgrade QEMU and libvirt.

• As required, follow the preceding step to upgrade QEMU and libvirt in all platform clusters in batches.
• Note: Disable the canary upgrade before upgrading.
• If QEMU cannot be upgraded in a customized or special version environment, this script cannot be used for the upgrade.
• After the QEMU upgrade, restart or migrate VM instances for the new version to take effect.

4. Post-upgrade environment check

• Check the canary upgrade switch.
• Check physical resource connections.
• Check VPC connections.
• Check service operation.
• After confirming that VM instance status is normal, enable high availability.

Verification

• Confirm that the platform version is 2.4.26 after the upgrade.
• Confirm that the QEMU and libvirt versions are updated.
• Confirm that physical resources, VPCs, VM instances, and services are normal.
• Confirm that the new QEMU version takes effect after VM instances are migrated or restarted.

Risks and rollback

• Risk: The management plane is unavailable during the online platform upgrade.
• Risk: Upgrading QEMU and libvirt may introduce compatibility issues.
• Risk: VM instances may need to be migrated or restarted for the upgrade to take effect.
• Risk level: Medium.
• Rollback: A complete dual-management-node rollback path is provided, including:
• Query the VIP and time source:

    zsha2 status
    zstack-ctl show_configuration | grep chrony.serverIp

• Decouple the dual management nodes:

    zsha2 stop-node
    systemctl disable keepalived
    systemctl disable zstack-ha
    mysql -uroot -pzstack.mysql.password zstack
    stop slave;
    reset slave all;
    show slave status;
    zstack-ctl config_ui --db-url jdbc:mysql://single_mn_ip:3306
    zstack-ctl configure DB.url=jdbc:mysql://single_mn_ip:3306
    zstack-ctl config_ui --webhook-host 127.0.0.1

• Delete and reinstall the platform:

    zstack-ctl stop
    cp /usr/local/zstack/dnsmasq /root/dnsmasq.bak
    rm -rf /usr/local/zstack
    bash /root/zstack-upgrade-bk -r -f ZStack_Cube-x86_64-DVD-2.2.30-c76.iso
    bash ZStack-Cube-installer-2.2.30.bin -D

• Restore the database and start the platform:

    zstack-ctl stop
    zstack-ctl restore_mysql -f /var/lib/zstack/mysql-backup/backup.gz --mysql-root-password zstack.mysql.password
    zstack-ctl start

• Reinstall dual-management-node high availability:

    ./zsha2 install-ha -nic br_bond0 -slave root:password@<slave_ip> -gateway <gateway> -vip <vip> -db-root-pw zstack.mysql.password -time-server <time_server> -force

Exception handling recommendations

If an exception occurs during the upgrade, immediately stop subsequent operations and recover the environment according to the rollback plan. If necessary, follow the dual-management-node decoupling, database recovery, and high availability reconstruction plan to roll back.