Issue/Introduction

This article troubleshoots snapshots that are not visible in ZSphere, show an empty UUID or 0 capacity, or cause a UI exception when deleted after ZMigrate, EISOO, or other backup software calls the single-volume snapshot API.

Symptoms

• A third-party task reports that it created or deleted a snapshot, but the ZSphere snapshot list has no corresponding record.
• The page shows an empty UUID and 0 capacity, and the UI must be refreshed after deletion is clicked.
• VolumeSnapshotVO can be queried through the CLI/database, while the page displays objects by snapshot group.
• When thin-provisioned data storage has insufficient capacity, deleting a single-volume snapshot might time out or report insufficient capacity.

Environment

• Product: ZVF ZSphere
• Version: 5.0.3
• Components: ZSphere single-volume snapshot API, snapshot-group UI, and data-storage capacity
• Deployment: Third-party software such as ZMigrate or EISOO calls the single-volume snapshot API

Possible causes

The current code handles APICreateVolumeSnapshotMsg and APICreateVolumeSnapshotGroupMsg separately. In the confirmed case, the third party called the former to create a single system-disk snapshot, while the ZSphere UI primarily manages snapshot groups. Therefore, a single-volume snapshot is not displayed as a normal snapshot group. An empty UUID or 0 capacity in the UI does not prove that the underlying snapshot does not exist.

Failure to delete a single-volume snapshot can be a separate branch: thin-provisioned data storage has PrimaryStorageCapacityVO.availableCapacity lower than the snapshot size, or CLI polling times out before the background merge finishes.

Resolution

1. Confirm whether the third party created a single-volume snapshot or snapshot group

• Check: Third-party task, called API, VM/disk UUID, snapshot UUID, and creation time.
• Location: Determine from third-party task logs whether CreateVolumeSnapshot or CreateVolumeSnapshotGroup was called, and query single-volume snapshots and snapshot groups separately through the CLI.
• Expected result: The third-party call and platform object type match. A snapshot group has a group UUID; a single-volume snapshot can be queried by volume or snapshot UUID.
• Exception handling: If only a single-volume snapshot exists and the UI displays only snapshot groups, the blank page is a display boundary; follow the single-volume branch. If the snapshot belongs to a group, do not use the single-volume deletion command in this article.

2. Check capacity and background tasks before deletion

• Check: PrimaryStorageCapacityVO.availableCapacity, target snapshot size, snapshot-deletion task, and KVM Agent logs.
• Location: Support staff perform read-only queries and compare availableCapacity > snapshot size. Also confirm that no deletion task for the same snapshot is still running:

1. From third-party and ZSphere task details, record the VM UUID, disk UUID, snapshot UUID, apiId, and exact time. Then run on the management node:

     KB_API_UUID='<apiId-from-task-details>'
     cd /usr/local/zstack/apache-tomcat/logs || exit 1
     ls -lh management-server.log*
     zgrep -a -n -C 3 "$KB_API_UUID" management-server.log*
     zgrep -a -n "$KB_API_UUID" management-server.log* | grep -iE 'json POST|http response|snapshot|delete|merge|timeout'

If the third-party task has no ZSphere apiId, use the snapshot UUID as the search value in the same zgrep, locate the request by exact time, and record its apiUuid, destination ZSphere host, and taskuuid/taskUuid.

2. On the ZSphere host addressed by the request, query current and archived KVM Agent logs by taskuuid:

     KB_TASK_UUID='<taskuuid-from-management-server-log>'
     cd /var/log/zstack || exit 1
     ls -lh zstack-kvmagent.log*
     zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*
     zgrep -a -n "$KB_TASK_UUID" zstack-kvmagent.log* | grep -iE 'snapshot|delete|merge|error|fail|timeout'

3. If the management-node request timed out but the KVM Agent still shows deletion or merge progress for the same taskuuid, continue observing the original task and do not submit deletion again. To inspect current logs, use vim -R LOG_FILE, enter /, paste the UUID, press Enter, use n for the next match, and enter :q! to quit.

• Expected result: Available capacity exceeds the snapshot size and no merge/deletion task is running for the target snapshot.
• Exception handling: Restore sufficient allocatable capacity first if capacity is insufficient. If the CLI times out while the KVM Agent log still shows deletion progress, continue observing the original task and do not resubmit. In the recorded case the CLI explicitly uses asynchronous polling; a frontend timeout does not mean background deletion failed.

3. Have the third party or support delete the exact single-volume snapshot

Workaround

• Applicability: Backend queries confirm that the third party created a single-volume snapshot rather than a snapshot group, deletion state and references can be uniquely identified, and the only current issue is that the UI list cannot display it or displays residue.
• Verified self-service method: Prefer having the third-party software that created the snapshot delete it through its original API/task chain, and record the returned snapshot UUID and result.
• Support-assisted method: If the third party reports successful deletion but the platform still returns the single-volume snapshot, ZStack Technical Support reconfirms the VM, volume, and snapshot UUIDs, absence of snapshot-group references, capacity, and maintenance window, and then runs through the CLI for the exact version:

  BatchDeleteVolumeSnapshot uuids=SNAPSHOT_UUID

• Expected result: The requested UUID matches the read-only query, the background deletion completes, and KVM Agent logs no longer show a merge for the snapshot.
• Exception handling: Do not infer a deletion target from an empty UI UUID or repeatedly submit by changing the global API timeout. If temporary overcommit adjustment is necessary because of insufficient capacity, record the original value, stop new allocations, ensure available capacity exceeds snapshot size, and restore the value immediately after deletion. Only support staff may perform this controlled operation.

4. Verify consistency among UI, object, and capacity

• Check: Single-volume snapshot query, snapshot-group query, data-storage capacity, and third-party task.
• Location: Confirm that the target single-volume snapshot is Deleted or absent and has no snapshot-group reference. Refresh the UI and query data-storage capacity again.
• Expected result: The UI no longer shows an abnormal entry with an empty UUID/0 capacity, capacity recovers after the background task finishes, and the next third-party task does not reference the deleted snapshot again.
• Exception handling: If the object is deleted but the UI remains abnormal, save the browser/API response and query results and escalate the UI display issue. If the object remains, continue troubleshooting background deletion and do not modify the database directly.

Additional information

• Support information must include the exact time window, resource and task UUIDs, complete error text, and relevant component logs.
• Do not extrapolate one site result to all versions. If the root cause remains open or an operation might affect services or data, stop self-service handling and submit the evidence listed in this article.