Scenario

• Product: ZCF
• Components: ZStack Cloud, ZStone
• ZStack Cloud version: 5.4.x
• ZStone version: 5.4.x
• Deployment: Dual management nodes
• Trigger: Primary-storage physical capacity is high and many Trash volumes have not been fully released.

Symptoms

• Primary-storage physical capacity is 89.83.x%.
• Virtual capacity is 31.41.x%.
• 796 rbd volumes remain in Trash and consume physical capacity.

Assessment

1. Verify that ceph -s is healthy.
2. Run rbd trash ls -p pool-xxx to inspect Trash volumes in the target pool.
3. After cleanup, check rbd trash ls and the storage UI to confirm that the volumes no longer exist.

Cause

In the ZStone V5 Trash mechanism, many rbd volumes were not fully released, continuously consuming primary-storage physical capacity and increasing the risk of a full pool.

Resolution

1. Preparation

• Risk level: Medium.
• Confirm that every target volume is in Trash before deletion.
• Deleted volumes cannot be rolled back.
• It is recommended to disable global high availability before the change.

2. Disable global high availability.

• Expected result: The platform enters controlled maintenance.

3. Monitor Ceph status:

   watch -n 2 ceph -s

• Expected result: The storage cluster remains healthy before and after cleanup.

4. Verify an individual volume:

   rbd trash ls -p pool-xxx | grep xxxx

• Expected result: The target volume is confirmed in Trash.

5. If an individual volume must be deleted forcibly:

   rbd trash remove -p pool-xxx xxx --force

• Expected result: The Trash volume is permanently removed.

6. Use the platform CLI to delete Trash volumes in batches:

   rbd trash ls -p pool-xxx

   zstack-cli

   LogInByAccount accountName=admin password=<account_password>
   CleanUpStorageTrashOnPrimaryStorage force=true uuid=xxx

The recommended primary method is to use the platform CLI for batch deletion; uuid=xxx is the target primary-storage UUID.

• Expected result: Trash volumes for the specified primary storage are cleaned in batches.

7. Check again:

   rbd trash ls -p pool-xxx

Also verify in the storage UI.

• Expected result: Trash volumes are gone and storage space is reclaimed.

Verification

• Verify that rbd trash ls -p pool-xxx is empty or significantly reduced.
• Verify that target Trash volumes have disappeared from the storage UI.
• Verify that primary-storage physical usage drops to a safe level.
• Verify that ceph -s remains healthy.

Risks and rollback

• Risk: Deleting an unintended volume causes irreversible data loss.
• Risk level: Medium.
• No rollback is available after deletion.