Document navigation

Applicable scenario

  • Product: ZCF
  • Component: ZCE-X
  • Version: 5.2.x
  • Trigger: Distributed storage capacity must be reduced by removing planned data disks.

Symptoms

  • The storage environment requires data-disk scale-in.
  • Scale-in triggers data rebalancing.
  • An incorrect sequence can cause I/O jitter, performance degradation, or more serious data risk.

Determination method

  1. Confirm ceph -s is healthy before scale-in.
  2. Confirm data recovery is complete and the cluster is healthy again after every removal round.
  3. Continue to the next disk or node only after the previous round is completely recovered.

Cause

This is ZCE-X storage scale-in. Remove disks one by one and recover after each removal to reduce data-rebalancing risk.

Resolution

  1. Preparation
  • Risk level: medium.
  • Minimize impact on normal VM services.
  • Confirm cluster health before every operation.
  • Process disks one by one in rounds; do not delete in parallel batches.
  1. Procedure
  2. Check storage-pool and cluster state:
   ceph -s
  • Expected result: Continue only when the cluster is healthy.
  1. Select Disable Data Recovery in the distributed-storage platform.
  • Expected result: Automatic recovery disturbance is reduced during the change.
  1. Go to Resource Management > Storage Pool, select the target storage pool, select Remove Disk, and remove planned disks from the pool.
  • Expected result: Target disks are removed from the storage pool.
  1. Go to Resource Management > Disk, select planned disks, and select Delete.
  • Expected result: Target disks are deleted from the platform.
  1. Enable data recovery and set it to medium-speed recovery.
  • Expected result: Data recovery begins while service impact remains acceptable.
  1. Wait for data recovery to complete and run:
   ceph -s
  • Expected result: The cluster returns to health.
  1. After the previous round fully recovers, repeat these steps for remaining data disks on the other two nodes.
  • Expected result: All scale-in actions complete as planned.

Verification

  • ceph -s returns to health before and after every round.
  • Target disks are removed from both the storage pool and platform.
  • Services have no obvious exception after data recovery completes.

Risks and rollback

  • Risk: Data rebalancing during scale-in can cause I/O jitter and reduced performance.
  • Risk: Insufficient replica count can cause data loss.
  • Risk: Insufficient remaining capacity can cause read-only state or write failures.
  • Risk level: medium.
  • Rollback: No separate detailed rollback command sequence is available. Reduce risk through strict round-by-round scale-in and health recovery.
Scale in a ZCE-X Ceph storage cluster | 技术方案 · KB | ZStack Resource Center