Deleting an abnormal empty storage pool from a ZStone cluster

Symptoms

• The ZStone storage cluster shows an abnormal status.
• ceph -s reports HEALTH_WARN.
• ceph osd pool ls returns three pools, one of which has a malformed name (missing the pool prefix).

Environment

• Product: ZCF
• Component: ZStone
• Trigger/Condition: a ZStone Ceph cluster reports HEALTH_WARN because of a malformed storage pool (in this case, a pool whose name was created without the pool prefix). The pool has no data and can be removed.

Diagnosis

1. Run ceph -s and confirm HEALTH_WARN.
2. Run ceph osd pool ls and identify the malformed pool.
3. Run ceph df and confirm that the malformed pool holds no data.

Cause

A storage pool with an incorrect name (missing the pool prefix) was created and is empty; it can be safely deleted to clear the cluster warning.

Resolution

1. Pre-Checks

• Send the change plan to the customer and obtain confirmation.
• Disable global HA on the cloud platform before performing the deletion.
• Back up the database on both management nodes
• zstack-ctl dump_mysql
• mkdir -p /root/zstack-db-backup
• cd /root/zstack-db-backup
• zstack-ctl dump_mysql --file-name zstack-db-backup-mn1-$(date +%F-%H%M%S).sql.gz
• ls -lh

2. Recovery Steps

1. After the customer confirms the change plan, disable global HA and back up the database on both management nodes.
2. Open a second terminal and run:

   watch -n 1 ceph -s

3. Capture the malformed pool's name and UUID from ceph df.
4. Delete the pool with ceph:

   ceph osd pool delete <UUID> --yes-i-really-really-mean-it

5. After the deletion completes, verify:

• The deletion reports success.
• ceph -s shows a healthy cluster status.
• The UI shows the cluster status as healthy.

6. Re-enable global HA on the cloud platform.

Verification

• ceph -s returns HEALTH_OK.
• ceph osd pool ls no longer lists the malformed pool.
• The ZStone UI shows the cluster status as healthy.
• VMs and workloads on the cluster remain unaffected.

Risks and rollback

• Deleting the wrong pool can cause data loss. Verify with ceph df that the target pool holds no data before issuing the delete, and use the correct UUID.
• The operation should be performed while global HA is disabled so the cloud platform does not react to the temporary cluster warning.
• Best practice: schedule periodic inspections to catch malformed or empty pools early.