Applicable scenario
- Product:
ZCF - Component:
ZCE-X - Version:
5.2.x - Trigger: Distributed storage capacity must be reduced by removing planned data disks.
Symptoms
- The storage environment requires data-disk scale-in.
- Scale-in triggers data rebalancing.
- An incorrect sequence can cause I/O jitter, performance degradation, or more serious data risk.
Determination method
- Confirm
ceph -sis healthy before scale-in. - Confirm data recovery is complete and the cluster is healthy again after every removal round.
- Continue to the next disk or node only after the previous round is completely recovered.
Cause
This is ZCE-X storage scale-in. Remove disks one by one and recover after each removal to reduce data-rebalancing risk.
Resolution
- Preparation
- Risk level: medium.
- Minimize impact on normal VM services.
- Confirm cluster health before every operation.
- Process disks one by one in rounds; do not delete in parallel batches.
- Procedure
- Check storage-pool and cluster state:
ceph -s
- Expected result: Continue only when the cluster is healthy.
- Select Disable Data Recovery in the distributed-storage platform.
- Expected result: Automatic recovery disturbance is reduced during the change.
- Go to Resource Management > Storage Pool, select the target storage pool, select Remove Disk, and remove planned disks from the pool.
- Expected result: Target disks are removed from the storage pool.
- Go to Resource Management > Disk, select planned disks, and select Delete.
- Expected result: Target disks are deleted from the platform.
- Enable data recovery and set it to medium-speed recovery.
- Expected result: Data recovery begins while service impact remains acceptable.
- Wait for data recovery to complete and run:
ceph -s
- Expected result: The cluster returns to health.
- After the previous round fully recovers, repeat these steps for remaining data disks on the other two nodes.
- Expected result: All scale-in actions complete as planned.
Verification
ceph -sreturns to health before and after every round.- Target disks are removed from both the storage pool and platform.
- Services have no obvious exception after data recovery completes.
Risks and rollback
- Risk: Data rebalancing during scale-in can cause I/O jitter and reduced performance.
- Risk: Insufficient replica count can cause data loss.
- Risk: Insufficient remaining capacity can cause read-only state or write failures.
- Risk level: medium.
- Rollback: No separate detailed rollback command sequence is available. Reduce risk through strict round-by-round scale-in and health recovery.
