Applicable scenario

• Product: ZCF
• Component: ZCE-X
• Versions: 4.4.x, 4.2.x, 4.x
• Operating system: C76
• Environment: dual management nodes
• Trigger: Cache-disk performance is abnormal, preventing associated OSDs from starting; replace the cache disk.

Symptoms

• A cache disk is abnormal.
• OSDs associated with that cache disk cannot start.
• Replace the cache disk and rebuild OSDs on the failed physical host.

Determination method

1. Confirm the issue is cache-disk performance abnormality.
2. Confirm affected OSDs are offline and must be removed from the storage pool.
3. After replacement, check that the new cache disk, OSDs, storage pool, and Ceph state return to normal.

Cause

The cache-disk performance abnormality prevents associated OSDs from starting. Remove the affected OSDs and cache disk, add a new cache disk, and rebuild the OSDs.

Resolution

1. Preparation

• Risk level: low.
• Estimated duration: approximately 1 hour.
• Change only the failed physical host; other VM services are not affected.
• Disable global HA and select Disable Data Recovery on the storage side.

2. Procedure
3. Disable global HA in ZStack Cloud.

• Expected result: The platform is ready for change.

2. Enable the Disable Data Recovery option for ZCE-X storage.

• Expected result: I/O fluctuations from data rebalancing are reduced during the change.

3. Process each OSD to be deleted on the affected node.

• Operate on OSDs from only one node at a time.
• For each OSD, run:

     echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
     cat /var/lib/ceph/osd/ceph-{osd_id}/type
     systemctl stop ceph-osd@{osd_id}
     systemctl status ceph-osd@{osd_id}

• After confirming that the cluster has no PGs in inactive / peering / unknown / unfound, remove the disk in the UI.
• Expected result: Target OSDs are stopped safely.

4. Remove associated OSD data disks from the storage pool.

• Expected result: OSD data disks are removed successfully.

5. Delete OSD data disks.

• After deletion, confirm cache-disk partition usage is 0/N.
• Expected result: No data disk uses the cache disk.

6. Delete the SSD cache disk.

• Expected result: The old cache disk is removed from the platform logic layer.

7. Confirm physical location and replace the cache disk.

• If the system does not recognize the new disk, manually configure disk passthrough.
• Expected result: The system recognizes the new cache disk.

8. Add the new SSD cache disk.

• Set the partition count according to actual requirements.
• Expected result: The new cache disk is added successfully.

9. Add OSD data disks.

• Add data disks in Disk Management.
• Enable cache and retain default memory read cache and cache mode.
• Wait for state to become Healthy.
• Expected result: New OSD data disks are created.

10. Add OSD data disks back to the original storage pool.

• Expected result: The storage pool accepts the new OSDs.

11. Enable data recovery.

• Wait until cluster data recovery reaches 100%.
• Expected result: Data recovery completes.

12. Check storage state.

• Expected result: The storage environment returns to normal.

13. Restore ZStack Cloud.

• Enable maintenance-mode physical hosts.
• Set global HA to True.
• Expected result: The platform resumes normal operation.

Verification

• The new cache disk is recognized and added.
• OSD state is Healthy.
• The storage pool accepts the disks.
• Ceph data recovery completes and remains normal.
• Physical hosts reconnect successfully.

Risks and rollback

• Risk: An incorrect disk-replacement, OSD-deletion, or recovery sequence can affect data security.
• Risk: Uncontrolled QoS during data recovery can cause service I/O fluctuations.
• Risk level: low.
• Rollback: If the new disk is abnormal, return the original disk to its original slot, add the disk, OSD, and storage pool using the original partitioning, set low-speed QoS for recovery, and restore HA only after ceph -s reports OK.