Applicable scenario

• Product: ZCF
• Component: ZCE-X
• Version: 4.2.x
• Deployment topology: dual management nodes
• Trigger: A two-replica storage pool contains multiple abnormal OSDs that were temporarily marked lost; after cluster health recovers, replace the isolated failed data disks one by one.

Symptoms

• Disks on multiple nodes are simultaneously in error state, causing OSD exceptions.
• Abnormal OSDs were temporarily marked lost, and the storage pool is currently healthy.
• Hardware-level disk replacement must still be completed.
• Note: Replace only one failed OSD on one node at a time.

Determination method

1. Before replacement, confirm ceph -s is healthy.
2. Confirm PostgreSQL primary-secondary replication is normal, LSN values match, and Diff=0.
3. Confirm ES and ETCD clusters are healthy.
4. After replacement, wait for data recovery to reach 100% before proceeding to the next disk.

Cause

The storage pool is not currently unavailable because abnormal OSDs were isolated temporarily, but failed underlying disks must still be replaced in a controlled sequence. Parallel operation across multiple nodes in limited-replica storage significantly increases data risk, so operate on one node and one OSD at a time and restore step by step.

Resolution

1. Preparation

• Risk level: medium.
• It is recommended to perform the operation during a low service-load period.
• Disable global cloud-platform HA.
• Back up databases on both management nodes.

2. Health checks and database backup
3. Back up databases on both management nodes:

   zstack-ctl dump_mysql --file-name zstack-db-backup-master
   zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Expected result: A database rollback baseline is available.

2. Check storage-cluster state:

   ceph -s

• Expected result: The cluster is ok.

3. Check PostgreSQL synchronization:

   xms-manage db show-replication-info

• Expected result: Primary-secondary LSN values match and Diff=0.

4. Check ES and ETCD health:

   xms-manage -s

• Expected result: Management dependencies are healthy.

3. Disable data recovery and observe state
4. Enable Disable Data Recovery in the upper-right corner of the storage platform.

• Expected result: The replacement procedure does not immediately trigger recovery pressure.

2. Continuously observe in a new window:

   watch -n 1 ceph -s

• Expected result: ceph state changes are available in real time.

4. Stop an OSD on one node and remove it from the storage pool

1. On the affected node, set the target OSD type to xstore3:

   echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
   cat /var/lib/ceph/osd/ceph-{osd_id}/type

• Expected result: The type marker is written correctly.

2. Stop the target OSD:

   systemctl stop ceph-osd@{osd_id}
   systemctl status ceph-osd@{osd_id}

• Expected result: The OSD is offline.

3. After no peering state remains in ceph, select Remove Disk in the storage pool.

• Expected result: The target OSD data disk is logically removed from the storage pool.

4. In Disk Management, select the OSD, confirm that its storage pool shows None, then delete it.

• Expected result: The old OSD record is deleted from the platform.

5. Hot-swap the disk and add it back
6. Hot-swap the failed disk at the hardware layer.
7. Confirm the host recognizes the new disk:

   lsblk

• Expected result: The new disk is recognized normally.

3. Select Add Disk in Disk Management.
4. Select the new data disk, enable cache as in the original configuration, and retain the original read-cache setting.

• Expected result: The new OSD data disk is managed correctly.

5. Wait until disk state becomes Healthy.
6. In the storage pool, select Add Disk to add the new OSD data disk back to the original pool.

• Expected result: The new OSD returns to the original storage pool.

6. Enable data recovery and complete
7. Select Enable Data Recovery in the console.

• Expected result: The storage pool begins data recovery.

2. Wait for cluster data recovery to reach 100%.
3. Check again:

   ceph -s

• Expected result: Storage state returns to OK.

4. In cloud-platform global settings, restore the VM HA switch to True.

• Expected result: Normal HA policy returns.

5. Proceed to the next failed-disk replacement only after all preceding steps complete and the state is normal.

Verification

• ceph -s is healthy.
• PostgreSQL, ES, and ETCD health states have no exception.
• The new disk state is Healthy after addition.
• Storage-pool recovery reaches 100%.
• Cloud-platform HA is restored.

Risks and rollback

• Risk: Removing a normal disk by mistake directly creates data risk.
• Risk: Replacing disks in parallel across multiple nodes amplifies two-replica storage-pool risk.
• Risk: Excessive recovery speed can increase service I/O latency.
• Risk level: medium.
• Rollback: “Disk replacement does not require rollback”; control risk through strict one-disk, one-node, stepwise verification.