Applicable Scenario

• Product: ZCF
• Component: ZStack Cloud
• Versions: 4.8.x, 5.4.x, and 4.18.x
• Current versions:
• Cloud 4.8.x
• Libvirt 8.0.x
• Qemu 6.2.x
• Kernel 4.18.0-425...
• Target versions:
• Cloud 5.4.x
• qemu-kvm-6.2.x
• libvirt-8.0.x
• Kernel 4.18.0-553.69.1.13...
• Deployment: dual management nodes and SAN storage

Symptoms

• The current platform and virtualization-component versions are outdated.
• The platform, qemu/libvirt, and host kernel versions need to be upgraded.
• More useful diagnostic logs are also required when a host crashes.

Diagnostic Procedure

1. Before the upgrade, use the upgrade-check script to verify environment health.
2. After the upgrade, verify that the platform, Agent, virtualization-component, and kernel versions match the target values.
3. Verify that workloads and VM networking operate normally.

Cause

This is a combined upgrade involving the platform, host crash parameters, and host kernels. Perform the work in stages and validate each host before proceeding.

Resolution

1. Configure host crash parameters.
2. Use config_crashdump.sh to configure crashdump.

• Purpose: produce more useful logs when a host crashes.
• Prerequisite: restart the host for the configuration to take effect.

2. Verify the configuration:

   systemctl status kdump
   dmesg | grep -i crash

• High-risk test:

     echo 1 > /proc/sys/kernel/sysrq
     echo c > /proc/sysrq-trigger

• After restart, check:

     ls -l /var/crash/

• Expected result: kdump and crash-retention settings are correctly configured.

2. Upgrade ZStack Cloud to 5.4.x.
3. Run zstack-upgrade-check-master to perform upgrade readiness checks.

• Expected result: the environment is healthy and upgrade risks are identified.

2. Disable global high availability.
3. Reserve one non-critical host for early reconnection verification.
4. On both management nodes, upload the ISO, upgrade script, HA package, and bin file, and verify the MD5 and commit values.
5. Adjust global CLI configuration:

   enable.multipath=ignore
   connect.timeout=1200
   update.os.parallelismDegree cluster=5/10
   update.os.parallelismDegree host=10

6. Stop all ZStack Cloud backup tasks.
7. Back up the database on both management nodes:

   zstack-ctl dump_mysql --file-name 4.x-mn1
   zstack-ctl dump_mysql --file-name 4.x-mn2

8. Update the repositories on both management nodes.
9. Identify the VIP node, extract and authorize the HA package, and stop the MN service.
10. On the VIP node, run:

    ./zsha2 upgrade-ha -gencfg=true
    zsha2 upgrade-mn -grayscale -yes -db-root-pw $mysql_root_password -peerpass $peer_root_password $zstack_bin

11. After the upgrade, check:

    zstack-ctl status
    zsha2 status

12. Log in to the UI and verify resource status.
13. Manually reconnect one non-critical host first, then verify kvmagent, VM operation, and workloads.
14. Check the Agent version:

    zstack-cli QueryAgentVersion

15. In SharedBlock or SAN environments, also check:

• vgs
• lsblk
• multipath
• I/O error in /var/log/messages
• dmesg

16. If no issue is found, reconnect the remaining hosts in sequence.
17. After the global upgrade is complete, disable the grayscale variable:

    UpdateGlobalConfig name=grayscaleUpgrade category=upgradeControl value=false

18. Enable global high availability.

• Expected result: the platform upgrade is complete and workloads remain stable.

3. Upgrade virtualization components.
4. Upgrade the following components on a single host or cluster:

• qemu-storage-daemon
• qemu-kvm
• glib2
• libvirt

2. Use QueryLongJob to check upgrade progress.
3. Use SSH to check versions in batches.
4. Run:

   virsh qemu-monitor-command $VMuuid --hmp info version

Verify the actual qemu version used by the VM.

5. After qemu is upgraded, restart or migrate the VM before the new version takes effect.

• Expected result: the virtualization-component upgrade takes effect.

4. Upgrade host kernels to 553.
5. Log in to ZStack Cloud and verify that the environment is healthy.
6. Disable global high availability and back up the database on both management nodes.
7. Verify that all VMs have been migrated away from the host to be upgraded:

   virsh list

4. Put the host into maintenance mode.
5. Extract the kernel-upgrade package:

   tar -xvzf upgrade_kernel_bcache.tar.gz

6. Run the upgrade:

   cd upgrade_kernel_bcache/
   bash upgrade_kernel.sh <host_ip>

7. Restart the host after the upgrade so that the new kernel takes effect.
8. Kernel package MD5:

   b327d608fd85d5afd27d906b09fb89ca

9. Verify the kernel version after the restart.
10. After verification succeeds, upgrade the remaining nodes in sequence.
11. After ZStack Cloud operates normally, re-enable global high availability.

• Expected result: all target hosts have been upgraded to kernel 553.

Verification

• Verify that the platform, Agent, qemu/libvirt/glib2, and kernel versions match the target values.
• Verify that SAN and SharedBlock status is normal.
• Verify that VM networking and workloads operate normally.

Risks and Rollback

• Risk: host kernel upgrades require restarts and can affect compatibility and workload continuity. Isolate each host before the upgrade.
• Risk: if VMs are not restarted or migrated after the qemu upgrade, they continue to run with the previous qemu version.
• Risk level: medium.
• Rollback: no complete rollback procedure is available. Reduce risk through staged execution and validation.