Issue/Introduction

This article applies when VM instance cloning fails, VM instance image creation fails, a snapshot cannot be deleted, the snapshot quota is reached, a snapshot chain is too long, or an LVM lock conflicts. These issues frequently occur during cloning, image creation, expansion, reset, snapshot rollback, or a snapshot API call from third-party backup software.

Symptoms

• Creating a root volume image fails and reports that snapshot.volume.num exceeds the quota.
• Cloning a VM instance fails and reports that a template cannot be created from a volume snapshot.
• Creating a system image from a VM instance fails and reports Data cannot be uploaded from local storage to the image repository.
• A batch snapshot deletion API call reports LV is already locked with incompatible mode.
• A snapshot chain is too long, snapshot associations are abnormal, or the management plane and data plane display inconsistent information.

Environment

• Product: ZCF
• Component: Cloud
• Affected objects: ZStack Cloud; VM instance, volume, snapshot, cloning, image creation, ImageStore, LVM, and Ceph or ZCE
• Version: 4.x-4.8.38

Possible causes

• A volume or VM instance has too many snapshots, reaching the account quota or causing a merge, upload, or indexing operation to time out.
• Cloning, image creation, expansion, reset, snapshot rollback, or another operation creates automatic snapshots that are not cleaned over time.
• A VM instance has a long snapshot chain, causing ImageStore Hash calculation, chunk upload, or metadata generation to fail.
• The image storage type does not match primary storage, or the available capacity of image storage is less than the volume capacity, snapshot chain capacity, or actual thick-provisioned volume requirement, causing prerequisite image creation for cloning to fail.
• Local storage, LVM, or zsblk automatic locking runs concurrently with offline snapshot deletion and causes a lock mode conflict.
• A task initiated through the snapshot API by third-party backup software times out or rolls back abnormally, leaving a snapshot, disabling the Delete button, or causing inconsistent management-plane and data-plane status.
• The volume XML, UI management plane, actual primary storage files, or backing chain is inconsistent.

Resolution

1. Identify the failed operation and object

• Record the VM instance UUID, volume UUID, snapshot UUID, account UUID, image storage UUID, primary storage type, and failure time.
• Determine whether cloning, system image creation, volume image creation, snapshot deletion, or a snapshot call from backup software failed.
• Check whether the task error contains a snapshot quota, ImageStore upload, volume snapshot, LVM lock, or backing file message.

2. Check the snapshot count, quota, and automatic snapshots

zstack-cli
QueryVolumeSnapshot volumeUuid=<volume-uuid>
QueryQuota identityUuid=<account-uuid>
QueryGlobalConfig category=primaryStorage name=undo.tempSnapshot

• If the account snapshot count reaches the limit, clean snapshots that are no longer required or adjust the quota before retrying image creation or cloning.
• To reduce one-time automatic snapshots created by image creation and full cloning, you can assess the primaryStorage.undo.tempSnapshot global setting based on current-version capabilities.
• Before cleaning snapshots, confirm the workload recovery boundary to avoid deleting a snapshot that a backup, clone chain, or rollback process still depends on.

3. Check the snapshot chain and image creation path

qemu-img info -backing-chain <volume-file>
grep -a -n "<snapshot-uuid-or-volume-uuid>" /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -a -n "<snapshot-uuid-or-volume-uuid>" /var/log/zstack/zstack-kvmagent.log*

• If a long snapshot chain exists, first assess deleting unnecessary snapshots, merging snapshots, or using a new full backup or image creation process.
• If an ImageStore upload fails, check the primary storage file, image storage capacity, imf2 metadata, and network transfer logs together.

4. Check LVM locks and storage-side status

lvmlockctl -i
journalctl -u lvmlockd --since "<start-time>" --until "<end-time>"
grep -a -n "LV is already locked\|lock" /var/log/zstack/zsblk-agent.log*

• For an LVM lock conflict, avoid concurrently retrying deletion of the same snapshot. First confirm the lock status, operation history, and background tasks.
• In a scenario where offline root snapshot deletion encounters automatic zsblk locking, the temporary workaround is to wait for the lock to be released before deleting again.

Detailed procedures

1. Snapshot count or quota exceeded

Applicable symptoms:

• Creating a root volume image fails and reports that snapshot.volume.num exceeds the quota.
• Automatic snapshot creation before image creation or cloning fails.
• The snapshot count is significantly higher than required by the workload.

Procedure:

1. In the UI or CLI, confirm the current snapshot count for the target account, volume, and VM instance.
2. Confirm with the workload owner which snapshots must be retained. Do not directly delete a snapshot required by rollback, backup, or a clone chain.
3. For a snapshot that is no longer required, prioritize deletion through the UI. Use the CLI only if the UI operation is unavailable.
4. After deletion, query the quota and snapshot count again and confirm that Current Used is below the quota limit.
5. If the workload must retain the current snapshot count, an administrator can adjust the snapshot count quota according to actual capacity and management policy.
6. Run the image creation or cloning task again.

If the issue originates from a one-time automatic snapshot created by image creation or full cloning, you can assess enabling automatic temporary snapshot cleanup:

zstack-cli
QueryGlobalConfig category=primaryStorage name=undo.tempSnapshot
UpdateGlobalConfig category=primaryStorage name=undo.tempSnapshot value=true

Notes:

• This setting mainly reduces residual one-time automatic snapshots created by volume image creation and full cloning.
• Whether it applies to fast full cloning, linked cloning, or snapshot rollback depends on current-version capabilities.
• Before enabling it, confirm the environment version, cloning method, and workload rollback requirements.

2. A long snapshot chain causes cloning or image creation to fail

Applicable symptoms:

• Cloning fails and reports failed to create template from the volume snapshot.
• VM instance image or volume image creation fails even though primary storage and image storage basic services are normal.
• A volume has a multi-level snapshot chain or multiple historical snapshots for each volume.

Procedure:

1. First confirm whether the VM instance workload is normal and whether the current snapshot must remain a rollback point.
2. Query the target volume snapshot chain and confirm snapshot count, creation time, and parent-child relationships.
3. For snapshots that the workload owner confirms are no longer required, first delete older snapshots or snapshots with no workload value to shorten the chain.
4. If snapshot deletion takes a long time, avoid submitting deletion repeatedly. Wait for the current task to complete before processing the next layer.
5. After deletion, run qemu-img info -backing-chain again and confirm that the chain is shorter and the backing file is normal.
6. Run cloning or image creation again.

For a critical VM instance, it is recommended that you complete a workload backup or perform the operation during a maintenance window. Do not batch-delete snapshots of unconfirmed purpose merely to complete a cloning task.

3. Image storage capacity or type does not meet cloning requirements

Applicable symptoms:

• Cloning fails and reports that no available image storage can be found or an intermediate image cannot be created.
• The VM instance volumes are large, such as a thick-provisioned root volume or multiple large data volumes.
• The primary storage and image storage types do not match, or image storage capacity is less than the volume or snapshot chain requirement.

Procedure:

1. Confirm whether VM instance cloning must first create a temporary image on image storage.
2. Check the capacity of all VM instance volumes, snapshot chain capacity, and the available capacity of image storage.
3. Confirm whether the image storage type supports the current primary storage type. Incompatible image storage cannot be used as the target of the cloning task.
4. If image storage has insufficient capacity, clean unnecessary images, expand image storage, or add compatible image storage with sufficient capacity.
5. If snapshot capacity is included in the prerequisite check, clean unnecessary snapshots according to the long snapshot chain procedure and retry cloning.

4. Abnormal ImageStore upload or imf2 metadata

Applicable symptoms:

• Creating a system image reports Data cannot be uploaded from local storage to the image repository.
• The VM instance has a snapshot chain, and image creation requires chunking, Hash calculation, index generation, and upload to ImageStore.
• Errors are concentrated in ImageStore metadata, Hash, index, or upload stages.

Procedure:

1. Check image storage free space and network connectivity to exclude insufficient capacity and network interruption.
2. Use qemu-img info -backing-chain <volume-file> to confirm that the snapshot chain is complete.
3. Check whether an abnormal imf2 metadata file exists in the corresponding directory.
4. If imf2 metadata is confirmed abnormal, first back up the related imf2 files and have ZStack Technical Support confirm whether ImageStore metadata must be regenerated.
5. After repairing the metadata, create the image again and create a test VM instance from the new image to verify startup.

Notes:

• imf2 files and the ImageStore index are internal image repository data. It is not recommended that you delete them directly before confirming the path and backup.
• For a long snapshot chain, it is recommended that you first shorten the chain or create a clean template image again.

5. An LVM lock conflict causes snapshot deletion to fail

Applicable symptoms:

• Snapshot deletion or batch snapshot deletion reports LV is already locked with incompatible mode.
• The logs contain lvmlockd, automatic zsblk locking, deactive installPath failed, or related information.
• Multiple snapshot deletion, backup, migration, cloning, or storage tasks exist at the same time.

Procedure:

1. Stop submitting repeated deletion requests for the same snapshot to avoid worsening the lock conflict.
2. Check Task Center and confirm whether a cloning, migration, backup, snapshot merge, or deletion task exists.
3. Check the lock status and related logs:

lvmlockctl -i
journalctl -u lvmlockd --since "<start-time>" --until "<end-time>"
grep -a -n "LV is already locked\|deactive\|lock" /var/log/zstack/zsblk-agent.log*
grep -a -n "<snapshot-uuid-or-volume-uuid>" /var/log/zstack/zstack-kvmagent.log*

4. If a temporary lock conversion conflict is identified, wait for the lock to be released and then delete again.
5. If repeated retries still fail, collect lvmlockctl, management-server, zsblk-agent, and KVM agent logs and have ZStack Technical Support confirm whether the lock service or storage remnants must be handled.

Notes:

• It is not recommended that you directly kill lvmlockd or sanlock or manually delete an LV in a production environment.
• Before restarting a lock service, renaming an LV, or deleting an LV, you must confirm workload impact, backup, and rollback plan.

6. A snapshot remains, the Delete button is disabled, or the management and data planes are inconsistent

Applicable symptoms:

• The snapshot size is displayed as 0 and the Delete button is disabled.
• Third-party backup software times out while creating a snapshot group and leaves a snapshot after an abnormal rollback.
• The UI displays a snapshot, but the VM instance XML or underlying backing chain no longer contains it.
• Snapshot deletion reports that an LV, image, or data-plane object cannot be found.

Procedure:

1. First check the VM instance XML and volume backing chain to determine whether the snapshot is still in the current volume chain.

qemu-img info -backing-chain <volume-file>
virsh dumpxml <vm-uuid>

2. If the snapshot is still in the chain, do not directly clean its management-plane record. Use the normal snapshot deletion process and wait for the merge to complete.
3. If the snapshot is confirmed absent from the chain but remains in the UI or database, first try deleting the snapshot record through the CLI.
4. If CLI deletion succeeds, check whether the UI, Task Center, and volume snapshot list become consistent.
5. If a suspected residual LV or file still exists at the underlying layer, first rename or isolate it for observation instead of permanently deleting it immediately. After confirming that no workload references it during the observation period, have ZStack Technical Support clean it.
6. For snapshots left by third-party backup software, it is recommended that you periodically inspect snapshot names, creation times, and volumeUuid values and cross-check them against backup job records.

Notes:

• Database changes, LV renaming, LV deletion, and manual snapshot remnant cleanup are high-risk operations.
• This customer-facing KB article recommends only confirming evidence and using normal deletion. If normal deletion fails, contact ZStack Technical Support.

7. Third-party backup software leaves snapshots after snapshot API calls

Applicable symptoms:

• Backup software frequently creates and deletes snapshots.
• Snapshot deletion fails and reports batch delete volume snapshot error, deactive installPath failed, or a deletion timeout.
• The management plane contains a residual snapshot created according to the backup software naming convention.

Procedure:

1. Compare backup software task times with cloud platform task times and confirm whether the residual snapshot originated from a backup job.
2. Check management node and KVM agent logs at the failure time and confirm whether a host timed out while performing deactivation or merge.
3. Check KVM agent status, memory usage, and system resources on the related host.
4. If only a residual snapshot exists and the chain is safe, use the CLI to delete residual snapshots individually.
5. Confirm subsequent API calls with the backup software provider. Prioritize the product-recommended snapshot group API to avoid single-volume snapshot remnants causing inconsistent UI display and deletion paths.
6. For an environment with frequent remnants, establish periodic inspections for backup-job residual snapshots, KVM agent resource trends, and snapshot deletion task duration.

8. Verify cloning and image creation after repair

• After cleanup or repair, retry cloning or image creation on a test object first.
• Create a VM instance from the new image and verify startup, volumes, network, and workload availability.
• For a snapshot process triggered by backup software, verify the complete path of creating, attaching, detaching, and deleting the snapshot and deleting the temporary volume.

Precautions

• Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
• Before a high-risk operation such as a database change, snapshot record repair, LV rename or deletion, lock service restart, forced deletion, OSD or MON rebuild, pool deletion, service restart, or network change, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.
• This information provides troubleshooting guidance only and does not replace current environment logs, snapshot chain, XML, storage-side status, and workload verification.