Scenario

• Product: ZCF
• Component: ZStack Cloud
• ZStack Cloud version: 5.3.x
• Current kernels:
• 4.19.90-52.15.v2207.ky10.x86_64
• 4.19.90-52.15.v2207.ky10.aarch64
• Target kernels:
• 4.19.90-52.57.v2207.ky10.x86_64
• 4.19.90-52.57.v2207.ky10.aarch64
• Deployment: Dual management nodes
• Trigger: Host kernels must be upgraded one at a time while controlling workload and storage impact.

Symptoms

• The current kernel version is old and must be standardized to 4.19.90-52.57.
• The upgrade requires host restart.
• In ZCE-X or separate-deployment environments, data recovery and Ceph health require attention while storage hosts are upgraded.

Assessment

1. Before upgrade, confirm that the platform is healthy and the target host is evacuated.
2. After upgrade, verify the new kernel version on the host.
3. For storage hosts, wait for ceph -s to return to normal before upgrading the next host.

Cause

This is a host-level kernel upgrade, not a hot patch for a single service. Evacuate hosts, use maintenance mode, restart, and validate one node at a time. In ZCE-X environments, storage-host upgrades additionally require Ceph health validation.

Resolution

1. Preparation

• Risk level: High.
• Verify that the platform is healthy.
• Disable global high availability.
• Back up databases on both management nodes.
• Migrate VMs from the host to be upgraded.

2. Platform and host preparation
3. Back up databases:

      zstack-ctl dump_mysql --file-name zstack-db-backup-master
      zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Expected result: A platform rollback baseline is available.

2. Confirm that VMs on the target host have been migrated away.
3. Run on the target host:

      virsh list

• Expected result: No VM is running.

4. Put the host into maintenance mode.

• Expected result: Workloads are not scheduled back to the target host.

3. Perform the kernel upgrade
4. Download and extract the kernel upgrade package for the applicable architecture on a management node.

• Expected result: Upgrade materials are ready.

2. Enter the upgrade directory and run:

      cd upgrade_kernel_bcache/
      bash upgrade_kernel.sh <host_ip>

Replace <host_ip> with the target host address.

• Expected result: The target host kernel is upgraded.

3. For a separate-deployment environment when upgrading a storage host:

• Disable data recovery first.
• Upgrade and restart hosts one at a time.
• After each restart, wait for:

        ceph -s

to return to normal before proceeding.

• Expected result: Concurrent restarts do not amplify storage-upgrade risk.

4. Restart the host after the upgrade for the new kernel to take effect.

• Expected result: The node starts with the new kernel.

5. Exit maintenance mode after the system returns to normal.
6. Check the kernel version after restart.

• Expected result: The version changes to 4.19.90-52.57.

7. After one host is verified, process the remaining nodes in sequence.
8. Re-enable global high availability after the platform is normal.

• Expected result: The upgrade is complete.

Verification

• Verify the target kernel version after host restart.
• Verify normal platform resource connections and workloads.
• For storage hosts, verify that ceph -s returns to normal after every host.

Risks and rollback

• Risk: Kernel upgrade requires host restart and directly affects workload continuity on that host.
• Risk: Concurrent upgrades of ZCE-X or storage hosts can affect Ceph health and workload stability.
• Risk level: High.
• No individual rollback command is provided. Control risk through single-host validation, maintenance mode, and sequential execution.