Applicable Scenario

• Product: ZCF
• Component: ZStack Cloud
• Versions: 4.8.x and 5.2.x
• ZCE version: 5.2.x
• Operating system: Kylin V10 SP3
• Kernel: 4.19.90-52.15.v2207.ky10.x86_64
• RAID controller: 9460-8i (SAS3508)
• Trigger condition: OSDs are frequently isolated by ZCE following dummy io detection timeouts, and RAID-controller driver stability is suspected.

Symptoms

• OSDs are frequently isolated after dummy io detection timeouts.
• The issue is initially suspected to be related to hardware stability.
• The server vendor recommends updating the RAID-controller driver first and observing the result.

Diagnostic Procedure

1. Verify that the environment and workloads allow node-by-node maintenance.
2. Check the megaraid_sas driver version before and after the upgrade.
3. After each node restart, wait for the storage pool to return to a healthy state before proceeding to the next node.

Cause

This is not a direct platform software error. Frequent OSD dummy io timeouts, together with the vendor recommendation, indicate a possible RAID-controller driver stability issue. Upgrade megaraid_sas to validate and mitigate the issue.

Resolution

1. Preparation

• Risk level: low, but this is an underlying driver change and must be performed node by node.
• Disable global high availability.
• Migrate VMs away from the target node.
• Back up the database and driver files.
• Disable data recovery during maintenance.

2. Procedure
3. Disable global high availability and migrate VMs away from the target node.

• Expected result: the target node can be safely taken out of service for maintenance.

2. Back up the driver and initramfs:

   mkdir -p /root/raid_driver_bak/
   cp /lib/modules/4.19.90-52.15.v2207.ky10.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko /root/raid_driver_bak/megaraid_sas.ko.bak
   cp /boot/initramfs-4.19.90-52.15.v2207.ky10.x86_64.img /boot/initramfs-4.19.90-52.15.v2207.ky10.x86_64.img.bak

• Expected result: a basis for quickly rolling back the driver is available.

3. Upgrade the driver:

   rpm -Uvh kmod-megaraid_sas-07.726.02.00-1.ky10.x86_64.rpm

• Expected result: the new driver is installed.

4. Check the version after the upgrade:

   cat /sys/module/megaraid_sas/version
   modinfo megaraid_sas | grep -E "^version:"
   cat /sys/module/megaraid_sas/version

• Expected result: the module version shows the target version.

5. Disable data recovery on the storage side.

• Expected result: automatic recovery does not increase risk during the driver-upgrade restart.

6. If the node is a management node, switch management services first:

   zsha2 demote
   zsha2 stop-node

• Do not perform this step for a non-management node.
• Expected result: management services do not conflict with the node restart.

7. Restart the host:

   sync&&sync&&sync&&reboot

• Expected result: the new driver takes effect after startup.

8. Check the driver version again after the node restarts.

• Expected result: the newly loaded driver version is confirmed.

9. Enable data recovery and wait for the storage pool to return to a healthy state.

• Expected result: the cluster returns to a stable state.

10. Repeat the preceding steps for the other nodes in sequence.
11. Enable global high availability after all nodes have been updated.

• Expected result: the driver upgrade is complete.

Verification

• Verify that the megaraid_sas driver version is correct on every node.
• Verify that the storage pool returns to a healthy state after each operation.
• Verify that OSDs are no longer frequently isolated because of dummy io timeouts.
• Verify that workloads and ZStack Cloud resources are operating normally.

Risks and Rollback

• Risk: this underlying driver change can affect compatibility and workload continuity.
• Risk: operating multiple nodes concurrently can cause storage errors and workload interruption.
• Risk: data recovery must remain disabled during the restart.
• Risk level: low; however, treat this as a sensitive underlying change.
• Rollback:

1. Replace the current driver with the backed-up .ko file:

     cp /root/raid_driver_bak/megaraid_sas.ko.bak /lib/modules/4.19.90-52.15.v2207.ky10.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko

2. Rebuild module dependencies and the boot image:

     depmod -a
     dracut -f /boot/initramfs-4.19.90-52.15.v2207.ky10.x86_64.img 4.19.90-52.15.v2207.ky10.x86_64
     reboot

3. After the restart, check:

     cat /sys/module/megaraid_sas/version
     rpm -qa | grep megaraid_sas