Environment

• Product: ZCF
• Component: ZCE-X
• Affected objects: ZCE-X, ZStack Cloud, Ceph, Postgres, ES and ETCD, ant-queen, and isolated OSDs
• Versions: ZCE-X 6.4.100.3 and ZStack Cloud 5.3.28 kylin sp3
• Trigger condition: Isolated OSD disks in production must be recovered after confirming environment health and the data recovery policy.

Symptoms

• The environment contains isolated OSD disks.
• In this scenario, isolation can be removed from all 3 disks at once.
• Continuously monitor Ceph to avoid affecting workloads or data consistency.

Diagnosis

1. Confirm that the Ceph cluster is OK.
2. Confirm matching Postgres primary and secondary LSN values and Diff=0.
3. Confirm ES and ETCD health.
4. Run ant-queen manage dump-isolated-disk to obtain the WWIDs and count.
5. Confirm healthy data disks and resumed data balancing in the UI.

Cause

The environment contains isolated OSDs. Follow the standard isolation-removal procedure to recover them in production without affecting workloads or data consistency.

Resolution

1. Prerequisites

• Risk level: Medium
• Confirm cluster health.
• Disable cloud platform high availability and turn on Disable Data Recovery on the storage platform.
• Continuously monitor ceph -s.

2. Procedure

1. Check the cluster and cloud platform, and confirm that Ceph is OK.

• Expected result: The prerequisites are met.

2. Check Postgres replication.

   xms-manage db show-replication-info

• Requirement: Matching LSN values and Diff=0.
• Expected result: Replication is normal.

3. Check ES and ETCD health.

   xms-manage -s

• Expected result: Both clusters are healthy.

4. Monitor storage in another window.

   watch -n 1 ceph -s

• Expected result: Ceph changes are visible in real time.

5. Under System Settings > Global Settings on the storage platform, turn on Disable Data Recovery.

• Expected result: Automatic recovery pauses.

6. Identify the primary ant-queen node and isolated disks.

   ant-queen manage dump-princess
   ant-queen manage dump-isolated-disk

• dump-isolated-disk shows each wwid and the count.
• Expected result: The target WWID list is obtained.

7. Remove isolation on the primary ant-queen node.

   ant-queen manage unset-isolate --wwid <wwid>

• Isolation can be removed from all 3 disks in this scenario at once.
• Expected result: The target disks are no longer isolated.

8. Confirm healthy data disks in the UI.

• Expected result: The isolated disks return to normal.

9. Restore data balancing by setting recovery QoS to low speed and enabling data recovery in the storage UI.

• Expected result: Data balancing resumes.

10. Re-enable global VM high availability.

• Expected result: Normal high-availability status is restored.

Verification

• Monitor ceph -s and confirm normal storage status.
• Confirm matching LSN values and Diff=0.
• Confirm healthy data disks in the UI.
• Confirm resumed balancing and high availability.

Risks and rollback

• Risk: Removing isolation in an unhealthy environment or with an incorrect recovery policy may affect workloads and data consistency.
• Risk level: Medium.
• Rollback: Follow the standard procedure strictly. If an abnormality occurs, stop subsequent operations immediately and begin emergency handling.

Exception handling recommendations

If an abnormality occurs, stop subsequent operations and follow the rollback plan. Complete an on-site rollback if necessary.