Scenario
- Product: ZCF
- Component: ZStack Cloud
- Version: 5.5.x
- Current kernel: 4.18.0-425.19.2.4.g0fd3a169.el8.x86_64
- Target kernel: 4.18.0-553.69.1.13.gc220c6303994.el8.x86_64
- Deployment: Dual management nodes
- Trigger: Host kernels must be upgraded to 553, with related driver and environment validation.
Symptoms
- Current host kernel versions are old.
- Hosts must be upgraded to 4.18.0-553.69.1.13.
- The change also requires mlnx-ofed driver and post-upgrade firewall-rule checks.
Assessment
- Confirm that the target host is evacuated before upgrade.
- Verify the kernel, driver, and NIC state after upgrade.
- Check for abnormal drop rules and assess them with the XSKY environment.
Cause
This is a full procedure of host evacuation, kernel upgrade, driver update, and network and firewall validation. Skipping mlnx-ofed update or XSKY rule inspection can leave driver incompatibilities or networking risk.
Resolution
- Preparation
- Risk level: Medium.
- Verify that the platform is healthy.
- Disable global high availability.
- Back up databases on both management nodes.
- Evacuate and maintain the target host
- Back up databases:
~~~bash zstack-ctl dump_mysql --file-name zstack-db-backup-master zstack-ctl dump_mysql --file-name zstack-db-backup-slave ~~~
- Expected result: A platform rollback baseline is available.
- Confirm that VMs have migrated away from the target host.
- Expected result: The host carries no workload instance.
- Run:
~~~bash virsh list ~~~
- Expected result: No VM is running.
- Put the host into maintenance mode.
- Expected result: The platform does not schedule workloads to it.
- Upgrade the kernel
- Extract the upgrade package on a management node:
~~~bash tar -xvzf upgrade_kernel_bcache.tar.gz ~~~
- Expected result: The upgrade-script directory is available.
- Run the upgrade script:
~~~bash cd upgrade_kernel_bcache/ bash upgrade_kernel.sh <IP> ~~~
- Expected result: The target host kernel is upgraded.
- Restart the host for the new kernel to take effect.
- Expected result: The host starts with the target kernel.
- Update drivers and validate networking
- Update the applicable bin package under /opt/zstack-dvd/x86_64/$os/zs_drivers/mlnx-ofed/.
- Expected result: The driver matches the new kernel.
- Rebuild initramfs and restart:
~~~bash dracut -f ~~~
- Expected result: The new driver is included in the boot environment.
- Reload the driver after restart:
~~~bash /etc/init.d/openibd restart ~~~
- Expected result: The NIC driver is loaded successfully.
- If the NIC is down, bring it up manually.
- Expected result: The network link returns to normal.
- Check XSKY firewall conditions
- Check:
~~~bash iptables-save | grep drop ~~~
- Expected result: Abnormal drop rules, if any, are identified.
- If a drop rule exists, check:
~~~text /etc/docker/daemon.json ~~~ XSKY Docker configuration in newer releases can cause this issue. Stop and contact the XSKY side if an abnormality is found.
- Expected result: Storage-configuration compatibility is assessed.
- Process hosts sequentially
- After validating one host, repeat the procedure for remaining hosts.
- Enable global high availability after all hosts are upgraded and ZStack Cloud is normal.
- Expected result: Host-kernel upgrade is complete.
Verification
- Verify that each host runs the target kernel.
- Verify normal mlnx-ofed and openibd status.
- Verify normal NIC state.
- Verify that iptables-save | grep drop has no abnormal output, or the issue has been identified.
- Verify normal platform operation.
Risks and rollback
- Risk: Kernel and driver upgrades can affect compatibility and workload continuity.
- Risk: Failure to evacuate or maintain a host can directly affect running VMs.
- Risk: New XSKY Docker configuration can introduce additional firewall-rule issues.
- Risk level: Medium.
- No explicit command-level rollback procedure is available; use sequential validation, maintenance windows, and stop-on-issue control.