Scenario

• Product: ZCF
• Component: ZCE-X
• Version: 4.2.x
• Storage media: Hybrid disks / all-SSD / all-HDD
• Deployment: Dual management nodes
• Trigger: An existing Ceph storage pool must be scaled in and its disks replaced before a new storage pool is created according to the new plan.

Symptoms

• The existing storage pool must be scaled in.
• The procedure removes OSDs and their corresponding cache disks.
• After all legacy resources are removed, a new storage pool must be created and initialized.

Assessment

1. Before the change, verify that Ceph, database replication, ES, and ETCD are healthy.
2. Process only one data node at a time, and do not continue with the next node until recovery is complete.
3. After the new storage pool is created, verify that ceph -s and platform resources have returned to normal.

Cause

This is a combined change involving storage-pool scale-in, OSD removal, cache-disk removal, and new-pool reconstruction. Only one node can be processed at a time; otherwise, data loss or excessive rebalancing can occur.

Resolution

1. Preparation

• Risk level: Medium.
• Disable global VM high availability.
• Back up the database on both management nodes.
• Verify the status of Ceph, PostgreSQL, Elasticsearch, and ETCD.

2. Health check and database backup
3. Back up the database on each management node:

      zstack-ctl dump_mysql node1
      zstack-ctl dump_mysql node2

• Expected result: A rollback baseline is available.

2. Check database replication:

      xms-manage db show-replication-info

• Expected result: Primary-standby replication is normal.

3. Check Elasticsearch health:

      /opt/sds/installer/bin/sds-util es_curl localhost:9200/_cluster/health?pretty

• Expected result: The status is green.

4. Check ETCD health:

      /opt/sds/bin/etcdctl cluster-health

• Expected result: The command returns cluster is healthy.

5. Confirm that Ceph is OK.

• Expected result: The environment can enter the scale-in phase.

3. Remove OSDs from one node
4. In the storage UI, enable Disable Data Recovery.

• Expected result: Automatic recovery does not introduce fluctuations during scale-in.

2. Log in to the target node and mark the OSD to be removed:

      echo xstore3 > /var/lib/ceph/osd/ceph-{osd_id}/type
      cat /var/lib/ceph/osd/ceph-{osd_id}/type

• Expected result: The target OSD type is changed correctly.

3. Stop the OSD:

      systemctl stop ceph-osd@{osd_id}
      systemctl status ceph-osd@{osd_id}

• Expected result: The service status is inactive.

4. Confirm that ceph -s contains none of the following:

• inactive
• peering
• unknown
• unfound
• Expected result: The cluster can continue with storage-pool removal.

5. Remove the corresponding OSD disk from the storage pool in the UI.
6. Delete the removed data disk.
7. Delete the corresponding cache disk.

• Expected result: The target disks are fully removed from the legacy storage pool.

4. Recover one node before processing the next node
5. Remove disks from only one data node at a time.

• Expected result: Data risks from concurrent multi-node scale-in are avoided.

2. After completing the current node:

• Enable data recovery.
• Enable global platform high availability.
• Set the storage-pool recovery QoS to low speed.
• Expected result: The cluster enters a controlled recovery state.

3. Wait for recovery to complete and for the cluster to become healthy before processing the next node.

• Expected result: Scale-in is completed in stages.

5. Rebuild the storage pool after all legacy nodes are processed
6. Select the storage pool and create a new pool based on the required capacity and performance.

• Expected result: New storage-pool metadata is created.

2. Select and add cache partitions according to the storage plan.

• Expected result: The cache structure of the new pool is created.

3. Select disks and create data disks.

• Expected result: New-pool data disks are ready.

4. Initialize the newly created storage pool and select all nodes that must join it.

• Expected result: The new storage pool is initialized.

Verification

• Verify that ceph -s returns to normal after each scale-in operation.
• Verify that each scale-in operation processes only one node.
• Verify that the new storage pool is created and initialized successfully.
• Verify that global high availability and baseline health return to normal.

Risks and Notes

• Risk: Removing disks from multiple data nodes at the same time can cause data loss.
• Risk: If recovery QoS is not set to low speed, recovery can increase front-end workload bandwidth impact.
• Risk level: Medium.

Rollback

If a replacement disk is abnormal after the change:

1. Return the replacement disk to its original slot.
2. Add the disk again with its original partitioning.
3. Add the OSD.
4. Add the storage pool.
5. Set the storage-pool recovery QoS to low speed.
6. Monitor recovery until ceph -s returns OK.
7. Enable global high availability and complete baseline health checks.