Environment

• Product: ZCF
• Component: ZCE-X
• Affected objects: Dual management nodes, hyperconverged architecture, 1 cache disk per node, OSDs, and Ceph storage pools
• Version: 5.2.115.6
• Trigger condition: OSDs on a node are offline because of a cache or data disk failure, requiring replacement of 1 cache disk on 1 node.

Symptoms

• OSDs on the failed node are offline.
• A cache or data disk failure may be the cause.
• Remove the failed OSDs before replacing the cache disk to maintain data consistency and operational safety.

Diagnosis

1. Confirm that the environment is within scope and record versions, system architecture, operating system, deployment mode, and storage type.
2. Confirm that Ceph is OK.
3. Confirm healthy Postgres, ES, and ETCD status.
4. Operate on only 1 node at a time.

Cause

In this single-cache-disk scenario, OSDs are offline because of a cache or data disk failure. To avoid data and metadata exceptions caused by replacing the disk directly, take the failed OSDs offline and remove them by following the standard OSD removal procedure before replacing the cache disk and restoring data balancing.

Resolution

1. Prerequisites

• Risk level: Medium
• Follow the plan strictly to avoid affecting workloads.
• Change only 1 node at a time.
• Continuous monitoring of ceph -s is recommended throughout the procedure.

2. Procedure

1. Check the cluster and cloud platform.

   xms-manage db show-replication-info
   xms-manage -s
   watch -n 1 ceph -s

• Expected result: Postgres, ES, ETCD, and Ceph are healthy.

2. Disable global high availability and back up both databases.

   zstack-ctl dump_mysql --file-name zstack-db-backup-master
   zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Expected result: High availability is disabled and backups are complete.

3. Disable Data Recovery on the storage platform.

• Expected result: Recovery traffic stops.

4. Change each target OSD type and stop it.

   echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
   cat /var/lib/ceph/osd/ceph-{osd_id}/type
   systemctl stop ceph-osd@{osd_id}
   systemctl status ceph-osd@{osd_id}

• Expected result: The type is xstore3, and the OSD is offline.

5. Monitor ceph -s. After no peering state remains, select and remove the corresponding host disk from the storage pool, and then delete it under Disk Management.

• Expected result: The failed OSDs are removed.

6. Clear the cache partitions.

• Expected result: Old cache partitions are removed.

7. Hot-swap the cache disk. If hot swap is unsupported, power off first.

   sync&&sync&&poweroff

• Power on afterward and confirm the disk with lsblk.
• Expected result: The new cache disk is detected.

8. Add disks to the storage pool with the same read-cache setting as before.

• Expected result: The new cache disk joins the pool.

9. Set recovery QoS to low speed and enable data recovery in the storage UI.

• Expected result: Data balancing begins.

10. Re-enable global VM high availability.

• Expected result: Normal platform operation is restored.

Verification

• Monitor ceph -s and confirm cluster health.
• Confirm healthy Postgres, ES, and ETCD status.
• Confirm that the new cache disk is detected and added.
• Confirm balancing and high availability.

Risks and rollback

• Risk: Remove OSDs in the UI only after completing low-level changes.
• Risk: A peering state lasting more than 10 seconds may affect workloads and requires recovery by following the emergency handling procedure.
• Risk: Rebalancing consumes recovery bandwidth, so use low recovery QoS.
• Risk: Confirm sufficient remaining capacity for balancing.
• Rollback: No separate complete rollback is provided. Use backups and emergency handling to restore workloads.

Exception handling recommendations

If a PG is stuck in peering or remapped and interrupts workloads, run:

ceph -s
ceph pg dump | grep -E "peering|remapped"
ceph pg dump | grep {pg_id}
ceph osd down {osd_id}