Environment

• Product: ZCF
• Component: ZCE-X
• Affected objects: ZStack Cloud 5.3.28, dual management nodes, Ceph and OSD, and storage pool pool-1
• Version: 6.4.100.3
• Trigger condition: Hot-swap 1 failed OSD disk without affecting VM operation or data safety.

Symptoms

• A single OSD disk must be replaced.
• The correct slot must be verified to avoid read/write latency or operational risk.
• If the new disk is not detected after hot swap, VMs may need to be migrated or the physical machine restarted.

Diagnosis

1. Confirm that the environment is within scope and record versions, system architecture, operating system, deployment mode, and storage type.
2. Identify the correct OSD by activating its indicator or verifying its slot.
3. Confirm healthy ceph -s and platform status, and prepare to disable high availability and back up the databases.

Cause

Replace 1 failed OSD while maintaining VM operation and data safety. Remove and delete the OSD before hot-swapping the disk and adding it again.

Resolution

1. Prerequisites

• Risk level: Low
• Verify the correct OSD slot to avoid removing the wrong disk.
• Disable high availability, back up the databases, and use an off-peak window.
• Prepare to migrate VMs or restart the physical machine if the new disk is not detected.

2. Procedure

1. Check cluster and cloud platform status.

• Expected result: Platform and storage status are normal.

2. Disable global high availability and back up both databases.

   zstack-ctl dump_mysql --file-name zstack-db-backup-master
   zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Expected result: High availability is disabled and backups are complete.

3. Continuously monitor storage in another window.

   watch -n 1 ceph -s

• Expected result: Storage health is visible in real time.

4. Turn on Disable Data Recovery on the storage platform.

• Expected result: OSD removal does not immediately trigger recovery.

5. Remove and delete the OSD from the storage pool.

• Set its type to xstore3 and check the file.

   echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
   cat /var/lib/ceph/osd/ceph-{osd_id}/type
   systemctl stop ceph-osd@{osd_id}
   systemctl status ceph-osd@{osd_id}

• After confirming that ceph -s has no inactive, peering, unknown, or unfound state, remove the OSD in the UI.
• Expected result: The OSD is removed and deleted.

6. Eject and hot-swap the failed disk.

• In the UI, identify the failed OSD disk as sdX and replace the target disk identified as sdY.
• Expected result: The correct disk is replaced.

7. Add the new OSD under Resource Management > Disk Management by clicking Create.

• Expected result: The new OSD is detected and created.

8. Add the OSD to pool-1 with a memory read cache of 256M.

• Expected result: The disk is added to the pool.

9. Enable data balancing and check platform and storage health.

   zsha2 status
   zstack-ctl status
   ceph -s

• Re-enable global VM high availability afterward.
• Expected result: Platform and storage are healthy and high availability is restored.

Verification

• Monitor ceph -s and confirm no abnormal PG state.
• Confirm that the new OSD is in pool-1 and platform resources are normal.
• Run zsha2 status and zstack-ctl status and confirm normal operation.

Risks and rollback

• Risk: Removing the wrong OSD creates unpredictable data and service risks.
• Risk: Hot swap may cause read/write latency.
• Risk: Insufficient remaining capacity may affect balancing and recovery.
• Rollback: Verify the slot and OSD status. If necessary, reinsert the original disk, restore the original configuration, and contact technical support.