Scenario

• Product: ZCF
• Component: ZCE-X, ZStone
• Version: 5.2.x
• Deployment: Three-server hyperconverged cluster with dual management nodes
• Trigger: A cache disk on cube-4 has failed and cannot be identified by the system. Multiple associated OSDs are abnormal and the cache disk must be replaced.

Symptoms

• A cache disk on cube-4 has failed.
• The operating system cannot identify the cache disk.
• Five associated OSDs are abnormal.

Assessment

1. Use casadm -L to identify disks associated with the failed cache disk.
2. Use lsblk to identify the mapping between device names and OSD IDs.
3. After replacement, verify that the new cache disk, OSDs, storage pool, and cluster are healthy.

Cause

A cache disk on the affected node has failed and is not recognized, causing multiple OSDs to become abnormal. Remove the association between the cache disk and OSDs before replacing and rebuilding the cache disk.

Resolution

1. Preparation

• Risk level: Medium.
• Disable global platform high availability.
• Monitor ceph -s continuously on the node being changed.
• Enable Disable Data Recovery in the storage platform.

2. Disable global high availability.

• Expected result: No unexpected high-availability action occurs during the change.

3. Log in to the storage node and monitor storage status:

   watch -n 1 ceph -s

• Expected result: Storage health changes can be observed in real time.

4. Enable Disable Data Recovery in the storage platform.

• Expected result: Automatic recovery does not introduce additional changes during the procedure.

5. Identify failed cache disk associations:

   casadm -L
   lsblk

• Expected result: The mapping between the failed cache disk and OSD data disks is known.

6. Remove OSD data disks associated with the cache disk from the storage pool.

• On the Storage Pool page, select Remove Data Disk.
• Select all OSD data disks on the target server associated with the cache disk.
• Expected result: Associated OSDs are removed safely.

7. Delete the OSD data disks.

• On the Data Disk page, delete OSDs removed from the storage pool.
• For hybrid storage, delete the SSD cache partitions after deleting the OSDs.
• On the Disk page, select Clean Cache.
• Expected result: Legacy cache and OSD associations are cleared.

8. Locate and replace the failed SSD cache disk.

• Use Locate Disk on the Disk page to identify the slot.
• If the server supports hot swapping, replace the disk directly.
• Expected result: The cache disk is physically replaced.

9. Add the new SSD cache disk.

• Configure cache partitions in Disk Management, using the same number of partitions as before replacement.
• Expected result: The new cache disk is initialized.

10. Add OSD data disks.

• Create data disks on the Data Disk Management page and enable cache partitions.
• Expected result: OSD data disks associated with the new cache disk are recreated.

11. Add the OSD data disks back to the storage pool.

• Expected result: OSDs rejoin the storage pool.

12. Enable data recovery and wait for the cluster to become healthy.

• Expected result: Storage data is rebalanced and recovered.

13. Perform a storage health check.

• Expected result: The cluster is healthy.

14. Restore global platform high availability.

• Expected result: The platform resumes normal operation.

Verification

• Verify that the new cache disk is recognized and partitioned.
• Verify that OSDs are recreated and added to the storage pool.
• Verify that ceph -s and the storage platform report a healthy state.
• Verify that resources are normal after high availability is restored.

Risks and rollback

• Risk: Failing to remove associated data disks before cache-disk replacement can increase storage data risk.
• Risk: Data recovery can increase I/O latency and affect high-I/O VMs.
• Risk level: Medium.
• No standard command-level rollback path is available. If an issue occurs, stop the operation immediately and escalate for a follow-up plan.