Troubleshooting snapshot deletion when control-plane and data-plane chains are inconsistent

Issue/Introduction

This article documents how to resolve a volume-snapshot deletion failure (could not find image ... in chain) caused by an inconsistency between the management database (control plane) and the underlying libvirt/QCOW2 backing chain (data plane). The investigation walks the chain, compares the on-host state to the database, and removes the stale DB row that libvirt can no longer match.

Symptoms

• Deleting a volume snapshot fails with:

  Operation failed, because invalid argument: could not find image
  '/dev/f179d81a3fd748608be2d4da35a50c8d/ff42324c90c144768f444683c6fd21c8'
  beneath '/dev/f179d81a3fd748608be2d4da35a50c8d/6174c7db57ac4942bee35556097cf7e6'
  in chain for '/dev/f179d81a3fd748608be2d4da35a50c8d/6174c7db57ac4942bee35556097cf7e6'

• One historical snapshot volume (/dev/f179d81a3fd748608be2d4da35a50c8d/ff42324c90c144768f444683c6fd21c8 in the source case) was deleted on the underlying storage but the management database still references it.

Environment

• Product: ZStack Cloud (ZCF).
• Version: 4.x and 5.x.
• Component: Primary storage (the source used a Ceph-based or LVM-based deployment that exposes /dev/<vg>/<image> paths to libvirt), libvirt, qemu-img, management database (VolumeSnapshotVO).
• Deployment: any deployment with shared-block or LVM/Ceph primary storage where a snapshot was deleted on the host but the management DB still lists it.

Possible causes

• The historical snapshot volume was actually deleted successfully at the underlying storage layer, but libvirt reported a failure, leaving the management database in an inconsistent state (database row present, no backing file on the host).
• A backing file referenced by the chain no longer exists on the host, so libvirt cannot resolve the chain when the next snapshot is deleted.
• A previous audit timeout or aborted deletion task left the DB row but did not roll the host state back.

Resolution

1. Check the snapshot audit for a timed-out deletion

• Check: confirm a deletion task failed earlier with a libvirt timeout while the underlying snapshot was actually removed.
• Expected result: the audit log shows a snapshot deletion that timed out (suggesting the underlying storage already removed the volume).

2. Log in to the physical host that owns the affected VM and inspect the backing chain

• Identify the VM's host (the one running the affected libvirt domain).
• Inspect the cloud disk (6174c7db57ac4942bee35556097cf7e6 in the source case):

  qemu-img info --backing-chain /dev/f179d81a3fd748608be2d4da35a50c8d/6174c7db57ac4942bee35556097cf7e6

• Inspect the next snapshot up the chain (8eb6fba2d9bb4d80b805a9def3e78883 in the source case):

  qemu-img info --backing-chain /dev/f179d81a3fd748608be2d4da35a50c8d/8eb6fba2d9bb4d80b805a9def3e78883

• Inspect the snapshot that libvirt can no longer find (ff42324c90c144768f444683c6fd21c8 in the source case):

  qemu-img info --backing-chain /dev/f179d81a3fd748608be2d4da35a50c8d/ff42324c90c144768f444683c6fd21c8

• Expected result: the could not find image snapshot reports no backing file, while the other two resolve cleanly.
• If abnormal: if even the cloud disk itself cannot resolve, stop and escalate — the inconsistency is deeper than a single missing row.

3. Confirm whether the snapshot file still exists on the host

• Check LVM for the snapshot UUID:

  lvs | grep -i <snapshot-uuid>

• Inspect the candidate path:

  qemu-img info /dev/<vg>/<image>

• Expected result: lvs returns no row for the missing snapshot, and qemu-img info returns "No such file or directory" — confirming the underlying storage has been cleaned up.

4. Inspect the VM's libvirt XML for the recorded snapshot configuration

• Dump the VM domain XML for backup:

  virsh dumpxml <vm-uuid> > /root/uuid.xml.bak

• Expected result: an XML record that lists the chain as the database does.

5. Remove the stale VolumeSnapshotVO row from the management database

• Disable global HA, then back up both management nodes' databases:

  zstack-ctl dump_mysql --file-name zstack-db-backup-master   # primary node
  zstack-ctl dump_mysql --file-name zstack-db-backup-slave    # standby node

• Log in to the database and locate the stale row:

  mysql -uroot -p'<mysql-root-password>' zstack
  select * from VolumeSnapshotVO where uuid = '8eb6fba2d9bb4d80b805a9def3e78883';

• Verify the primaryStorageInstallPath column is empty — this confirms the underlying storage no longer references the row.
• Delete the stale row:

  delete from VolumeSnapshotVO where uuid = '8eb6fba2d9bb4d80b805a9def3e78883';

• Expected result: the row is gone; select returns zero rows.
• If abnormal: restore from the backup taken at the start of this step.

6. Retry the snapshot deletion

• UI: retry the original snapshot deletion that failed.
• Expected result: the deletion succeeds; the remaining snapshot chain is consistent.

7. Verify the result

After deleting the stale VolumeSnapshotVO row (Step 5), retry the snapshot deletion from the UI. The task should complete successfully and the snapshot should disappear from the snapshot list. The VM should remain in Running state throughout. Verify on the host with qemu-img info <backing-file> that the backing chain is consistent.

8. Record risks and rollback information

WARNING: Step 5 modifies the management database directly (delete from VolumeSnapshotVO). This operation is OUT-OF-BAND and is not exposed through any ZStack API. Only support engineers with database access should perform this step, and only after taking a full MySQL dump on both management nodes. Consider restricting this KB to visibility: partner or internal. Risk: deleting the wrong row will leave the database in an inconsistent state with the actual snapshot files on disk. Use the exact UUID from the audit log, not a substring match. Precaution: back up MySQL on both management nodes before the operation. Rollback: restore from the MySQL dump taken in precaution.

Additional information

Prevention and optimization

• Treat deletion timeouts as a likely "underlying-storage-succeeded-but-libvirt-failed" scenario — verify on the host before retrying.
• Always take a fresh DB backup of both management nodes before any direct DB modification.
• Avoid deleting snapshots in rapid succession on the same volume; chain validation by libvirt depends on a stable state.
• When a deletion fails, capture the full audit log and qemu-img info --backing-chain output before attempting recovery, to keep the audit trail.

Escalation to R&D

If after applying this procedure the snapshot deletion still fails with the same could not find image error, escalate to ZStack R&D with the following artifacts:

• management-server.log covering 30 minutes before and after the deletion attempt (from /var/log/zstack/management-server.log on the management node)
• qemu-img info --backing-chain output for the affected cloud disk and the surrounding snapshots
• virsh dumpxml <vm-uuid> output captured before the direct DB edit
• The VolumeSnapshotVO row that was deleted, with timestamps (from the audit log)
• A fresh zstack-ctl dump_mysql taken after the edit (do not include the management-node password in the dump file name)

Do not retry the direct DB edit on additional rows before R&D review. R&D may want to inspect the remaining snapshot chain before any further changes are made.

Supporting information

• Related components: Primary Storage (Ceph / LVM / Shared Block), libvirt, qemu-img, management DB VolumeSnapshotVO.
• Related commands: qemu-img info --backing-chain, virsh dumpxml, lvs, zstack-ctl dump_mysql, mysql.
• Disclaimer: direct DB edits are out-of-band operations; coordinate with ZStack support before performing step 5 in production.