Environment

• Product: ZCF
• Component: ZCE-X
• Affected objects: Dual management nodes, OSD disks, Ceph, and ZCE-X storage pools
• Version: 5.2.115.5
• Trigger condition: An isolated OSD disk has bad sectors and must be replaced individually.

Symptoms

• An OSD in the storage environment is isolated.
• Low-level checks find bad sectors on the disk.
• Hot-swap the disk and keep data recovery and balancing safe.

Diagnosis

1. Confirm that the environment is within scope and record versions, system architecture, operating system, deployment mode, and storage type.
2. Confirm that the Ceph cluster is healthy.
3. Run xms-manage db show-replication-info and xms-manage -s and confirm normal database, ES, and ETCD status.
4. Operate on only 1 OSD on 1 node at a time.

Cause

The target OSD is isolated and has bad sectors. Take it offline safely, hot-swap the disk, and use low-speed recovery to limit the impact on workloads.

Resolution

1. Prerequisites

• Risk level: Low
• Disable cloud platform high availability and back up the databases.
• Operate on only 1 OSD on 1 node at a time.
• Continuously monitor ceph -s and avoid prolonged peering or remapped states.

2. Procedure

1. Check the cluster and cloud platform.

   xms-manage db show-replication-info
   xms-manage -s
   watch -n 1 ceph -s

• Expected result: Database, ES, ETCD, and Ceph status are normal.

2. Disable global high availability and back up both databases.

   zstack-ctl dump_mysql --file-name zstack-db-backup-master
   zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Expected result: High availability is disabled and backups are complete.

3. Disable Data Recovery on the storage platform.

• Expected result: Recovery traffic stops.

4. Change the OSD type and stop the OSD from the command line.

   echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
   cat /var/lib/ceph/osd/ceph-{osd_id}/type
   systemctl stop ceph-osd@{osd_id}
   systemctl status ceph-osd@{osd_id}

• Expected result: The type is xstore3, and the service is stopped and offline.

5. Monitor ceph -s. After confirming no persistent peering, remove and delete the OSD from the storage platform.

• Expected result: The failed OSD is removed.

6. Hot-swap the offline failed disk.

• Expected result: The new disk is detected.

7. Add the new disk to the storage pool with the same read-cache configuration as the original environment.

• Expected result: The disk joins the pool.

8. Enable data balancing. Set recovery QoS to low speed in the storage UI and enable Data Recovery.

• Expected result: Low-speed recovery begins.

9. Enable global VM high availability.

• Expected result: Normal high-availability status is restored.

Verification

• Run ceph -s and confirm cluster health.
• Confirm that the new disk is detected and added to the pool.
• Confirm normal workload operation after high availability is restored.

Risks and rollback

• Risk: Removing an OSD in the UI before completing low-level processing may cause an abnormality.
• Risk: Prolonged peering or remapped states may affect workloads.
• Risk: Scale-in and recovery create rebalancing traffic, so keep recovery QoS low.
• Rollback: If a PG is stuck, identify it and run:

   ceph -s
   ceph pg dump | grep -E "peering|remapped"
   ceph pg dump | grep {pg_id}
   ceph osd down {osd_id}