Volume snapshot creation fails with "unable to find volume" due to a path mismatch

Symptoms

• Creating a snapshot of a volume fails with unable to find volume.
• The volume already has snapshots on it (i.e., this is not a brand-new volume).
• The path the database records for the volume does not match the actual qcow2 path on the host.

Environment

• Product: ZStack Cloud.
• Version: all versions specified by the source material.
• Component: volume snapshots, management database, and primary storage.

Diagnosis

• Confirm that the volume already has snapshots.
• Compare VolumeEO.primaryStorageUuid and VolumeEO.installPath with the live VM's libvirt XML and host-side storage path.

Resolution

1. Prerequisites and procedure

1. Confirm there are existing snapshots on the volume in the UI.

2. On the physical host where the VM runs, dump the live VM's libvirt XML to read the actual qcow2 path:

   # virsh dumpxml $vmuuid

Record the <source file="..."> entry under <disk> (that is, $device) and the corresponding path of the disk on the host.

3. Confirm that there are no running tasks for the cloud volume in the UI or on the host:

   # virsh blockjob $vmuuid $device

Replace $device with the disk target in the XML, and replace $vmuuid with the UUID of the running cloud VM.

4. Back up the database, disable cloud VM HA, and stop scheduled tasks:

   # zstack-ctl dump_mysql --file-name zstack-db-backup-master    # Run on the primary management node
   # zstack-ctl dump_mysql --file-name zstack-db-backup-slave     # Run on the standby management node

5. Modify the database

5.1 Log in to the database

   # mysql -u root -pzstack.mysql.password zstack

5.2 Query the database and confirm that the primaryStorageUuid and installPath of the cloud volume are inconsistent with the host-side path:

   select * from VolumeEO where uuid=$Volume-UUID;

$Volume-UUID is the UUID of the disk If the values in the database are inconsistent with the XML / host-side values, perform the following modification.

5.3 Modify:

   update VolumeEO set primaryStorageUuid="$Real-PS-UUID-From-Xml", installPath="$fixed-InstallPath-From-Xml" where uuid="$Volume-UUID";

Parameter description:

• $Real-PS-UUID-From-Xml is the primary storage UUID corresponding to the storage path in the XML.
• $fixed-InstallPath-From-Xml is the normalized XML storage path according to the primary storage type:
• For Shared Block primary storage, replace /dev/ with sharedblock://.
• For Ceph primary storage, add ceph:// before the pool.

= If you are unsure, refer to the installPath format of another normal cloud volume on the same primary storage.

6. Merge old snapshots

6.1 Log in to the ZStack Cloud UI and reduce the global setting "Maximum number of snapshot increments" to 2 so a full snapshot is forced sooner.

6.2 Create new snapshots consecutively; after at most two, a full snapshot is created. Adjust the snapshot timeout if needed.

6.3 Delete the older snapshots that previously failed.

7. Confirm in the UI that new snapshots can be created successfully.

8. Re-enable VM-level global high availability and check that platform resource connectivity is healthy.

Verification

• New volume snapshots can be created in the UI without error.
• Old snapshots can be deleted cleanly after the full snapshot is produced.

Cause

The management database's VolumeEO.installPath and VolumeEO.primaryStorageUuid are out of sync with the actual location of the volume on the host. The snapshot code path therefore cannot find the qcow2 file and returns unable to find volume.

Risks and rollback

• Back up the database and disable global VM HA before changing VolumeEO.
• If the update is incorrect, restore the database backup taken before the operation and revert the volume metadata to its original values.