Issue/Introduction

Use this article to troubleshoot a task that fails after a fixed period with live merging snapshot chain failed, block job not finished when deleting one single-volume snapshot, deleting multiple single-volume snapshots in a batch, or deleting a snapshot group. First identify the actual deletion entry point, then check the corresponding API timeout configuration and the deadline for the same task, and finally determine whether the underlying block job was canceled at its deadline or has a storage, capacity, or snapshot-chain issue.

Symptoms

• APIDeleteVolumeSnapshotMsg and APIBatchDeleteVolumeSnapshotMsg have been changed to 24 hours, but snapshot deletion still fails after about 3 hours.
• The management node log contains VolumeSnapshotTreeBase$35, batch delete volume snapshot error, or live merging snapshot chain failed, block job not finished.
• When a snapshot group is deleted, the outer DeleteVolumeSnapshotGroupAction in the task details might show success while results[].success for one member is false, followed by a runtime error.
• In the recorded case, the actual request was APIDeleteVolumeSnapshotGroupMsg. It carried timeout=10800000, or 3 hours. When the deadline was reached, the online full-rebase pull block job was still progressing and was then canceled.

Environment

• Product: ZVF ZSphere
• Version: 4.10.25.65 (recorded case; for other versions, first query the actual configuration as described here instead of directly applying this version-specific conclusion)
• Components: Single-volume snapshots, snapshot groups, management node API timeout, KVM Agent, and libvirt block jobs
• Deployment: Deleting a snapshot of a running VM requires an online snapshot-chain merge; the recorded case used a SharedBlock data store

Possible causes

The symptom can have one or more of the following causes:

• The changed configuration item does not match the actual operation entry point. For example, the site is deleting a snapshot group, but only the API timeouts for single and batch single-volume snapshot deletion were changed.
• The configuration was changed after the task was created. The deadline of an already-started request is fixed; changing GlobalConfig afterward does not extend the existing task.
• The timeout was extended correctly, but the underlying block job is progressing slowly or has stopped. Further checks might be required for storage I/O, iSCSI paths, data-store capacity, the snapshot chain, or QEMU/libvirt state.
• Only the outer success value of the snapshot-group event was checked. The results[] for each member was not checked, so a single-volume deletion failure was missed.

Do not treat the following errors as the root cause or a configuration name:

• VolumeSnapshotTreeBase$35 is where the upper-level workflow reports the error, not a GlobalConfig name.
• If disk vda does not have an active block job appears after the deadline was reached and the task was canceled, it is usually secondary information from another block-job cancellation attempt during cleanup. First locate the earlier timeout and merge-progress logs.

Resolution

1. Determine whether the operation deletes one single-volume snapshot, a batch of single-volume snapshots, or a snapshot group

• Check: Operation name, API ID, snapshot UUID, snapshot-group UUID, VM UUID, and start time in the task details.
• Location: Open the failed task details in the ZSphere UI and record the complete error, start and end times, and apiId. Then map the actual operation to the following table:

• Expected result: The task details and management node log identify which API the same failed request used.
• If abnormal: Seeing only VolumeSnapshotTreeBase$35 or "Delete Snapshot" on the page is not enough to select a configuration item. Continue to step 2 and confirm the actual API from the management node log.

2. Use apiId to query the management node request, timeout, and deadline

• Check: API class name, timeout, deadline, internal messages, and target ZSphere host for the same request.
• Location: Log in to the management node over SSH and use the apiId recorded in step 1 to query current and archived logs. Replace the example value in quotes with the site-specific value:

  KB_API_UUID='<apiId-from-task-details>'
  cd /usr/local/zstack/apache-tomcat/logs || exit 1
  ls -lh management-server.log*
  zgrep -a -n -C 5 "$KB_API_UUID" management-server.log*
  zgrep -a -n "$KB_API_UUID" management-server.log* | grep -iE 'APIDeleteVolumeSnapshot(Group)?Msg|APIBatchDeleteVolumeSnapshotMsg|DeleteVolumeSnapshotMsg|MergeVolumeSnapshotOnKvmMsg|timeout|deadline|mergesnapshot|block job|error'

• Expected result:
• Single-volume or batch single-volume deletion matches its respective API; snapshot-group deletion matches APIDeleteVolumeSnapshotGroupMsg.
• For a new request created with a 24-hour timeout, the log shows 86400000 milliseconds and a new deadline matching the request time.
• The internal DeleteVolumeSnapshotMsg, MergeVolumeSnapshotOnKvmMsg, and /vm/volume/mergesnapshot request sent to the KVM Agent belong to the same call chain, and their remaining time matches the upper-level request.
• If abnormal:
• If the actual entry point is snapshot-group deletion but the log still shows 10800000 milliseconds or an approximately 3-hour deadline, the wrong configuration item was changed. Continue to step 3.
• If the actual entry point and timeout are correct but the deadline is still the old value, confirm whether the task started before the configuration change. An existing task cannot obtain a new deadline from a later change.
• If you cannot find the Agent taskUuid by apiId, continue querying by snapshot UUID, snapshot-group UUID, VM UUID, and the exact time window, and save every matching line.

3. Query the current configuration for all three deletion entry points in the CLI

• Check: Actual GlobalConfig values for all three deletion entry points. Values can differ by version or environment, so query them before deciding whether to make a change.
• Location: Log in to the management node over SSH, enter zstack-cli, sign in with an authorized account, and run each command:

  QueryGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.APIDeleteVolumeSnapshotMsg
  QueryGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.APIBatchDeleteVolumeSnapshotMsg
  QueryGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.group.APIDeleteVolumeSnapshotGroupMsg

• Expected result: The configured value for the actual API identified in step 1 covers the expected deletion and merge time. Save all three original values in the support ticket.
• If abnormal: Do not query a similarly named configuration just because the error location contains VolumeSnapshotTreeBase$35. Do not blindly change all three configurations at once. Change only the item corresponding to the actual entry point, and continue checking the health of the underlying task.

4. Check whether the KVM Agent and libvirt block job are still progressing

• Check: Agent taskUuid, VM state, target disk, block-job progress, and snapshot-chain backing information.
• Location: First obtain the target ZSphere host, Agent taskUuid, and VM UUID from the management node log in step 2. Then log in to that ZSphere host over SSH. Use the result of virsh domblklist to identify the target name for the affected disk; vda is only an example below and must be replaced if the site's value differs:

  KB_TASK_UUID='<taskUuid-from-management-server-log>'
  KB_VM_UUID='<affected-vm-uuid>'
  KB_DISK_TARGET='vda'
  cd /var/log/zstack || exit 1
  ls -lh zstack-kvmagent.log*
  zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*
  zgrep -a -n "$KB_TASK_UUID" zstack-kvmagent.log* | grep -iE 'mergesnapshot|fullRebase|block|progress|timeout|deadline|cancel|error|fail'
  virsh domstate "$KB_VM_UUID"
  virsh domblklist "$KB_VM_UUID" --details
  virsh blockjob "$KB_VM_UUID" "$KB_DISK_TARGET" --info
  virsh qemu-monitor-command "$KB_VM_UUID" --pretty '{"execute":"query-block-jobs"}'

• Expected result:
• While deletion is running, the block job exists, the amount of processed data increases across two consecutive queries, and there are no storage-path, I/O, or capacity errors.
• After the task completes normally, the block job can be absent, but the management node and Agent logs contain successful responses, and the VM disk backing relationship matches the platform record.
• If abnormal:
• If virsh blockjob reports no task, first check chronologically whether a timeout and cancellation already occurred. Do not treat "no active block job" during cleanup as the initial cause.
• If the block job exists but makes no progress across consecutive observations, or if the log contains I/O, iSCSI-path, VG-capacity, or snapshot-chain errors, stop retrying, preserve the output, and contact ZStack Technical Support to investigate the underlying cause. Extending the API timeout does not fix these problems.
• Do not manually run blockJobAbort, delete an LV, modify snapshot state in the database, or restart the management node to clean up the site.

5. Check the result for each snapshot-group member

• Check: results[] in the APIDeleteVolumeSnapshotGroupEvent response, plus each member's snapshot UUID, volume UUID, success value, and error information.
• Location: Expand the complete response in the task details, API response, or management node log from step 2. Record whether each failed member is a root disk or data disk.
• Expected result: Snapshot-group deletion is successful only when every member in results[] succeeds.
• If abnormal: If the outer event has success=true but any member has results[n].success=false, treat the deletion as failed. Use the failed member's volume UUID, snapshot UUID, and Agent taskUuid to return to step 4 and locate the corresponding block job.

6. Extend the timeout for the correct entry point in a controlled window and submit a new task

Workaround

• Preconditions: The actual API is confirmed; the old deletion task has ended; no snapshot, backup, merge, or migration task is running for the same VM; VM state and disk backing chain are consistent; the data store/VG has sufficient space; step 4 found no I/O, path, or chain anomaly; the original GlobalConfig values are saved; and maintenance-window and business approval have been obtained.
• Location: In zstack-cli on the management node, run only the update command matching the actual entry point. The following 24-hour value comes from the recorded case and is a temporary observation window, not a fixed recommendation for every environment:

  # When deleting one single-volume snapshot
  UpdateGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.APIDeleteVolumeSnapshotMsg value=24h

  # When deleting multiple single-volume snapshots in a batch
  UpdateGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.APIBatchDeleteVolumeSnapshotMsg value=24h

  # When deleting a snapshot group
  UpdateGlobalConfig category=apiTimeout name=org.zstack.header.storage.snapshot.group.APIDeleteVolumeSnapshotGroupMsg value=24h

• Submit again: After changing the configuration, create a new deletion request. The existing request's deadline is fixed, and changing the configuration does not extend it. Immediately after the new request starts, repeat step 2 and confirm that the API request and Agent call carry 86400000 milliseconds and a matching new deadline.
• Monitoring: While the task runs, repeat the read-only queries in step 4 at fixed intervals to confirm that the block job continues to progress. Also monitor data-store capacity, host I/O, and business latency. For a snapshot group, check every results[] member as well.
• Success criteria: The new task finishes before its deadline; single-volume deletion succeeds, or every snapshot-group member succeeds; the corresponding block job no longer exists; VM operation, the disk backing chain, and the platform snapshot inventory are consistent; and no new I/O or capacity alert appears.
• Stop conditions: Immediately stop further retries and contact ZStack Technical Support if the block job makes no progress for an extended period, is unlikely to finish within the maintenance window, encounters a storage-path, I/O, capacity, or snapshot-chain error, causes a VM anomaly, or if snapshot-chain consistency cannot be confirmed. Do not conceal an underlying problem by repeatedly increasing the timeout.
• Rollback: After the task succeeds or fails and is confirmed to have completely ended, restore the changed GlobalConfig to the original value recorded in step 3, then run QueryGlobalConfig to confirm it. Restoring the original value affects only requests created afterward; it does not cancel or shorten a running task.

Additional information

• Include the following when submitting a support request: screenshot of the failed task; API ID; snapshot, snapshot-group, VM, and disk UUIDs; exact time window; query results for all three API timeouts; matching management node logs; Agent taskUuid logs; at least two block-job progress samples; data-store capacity; and abnormal alerts.
• The recorded case demonstrates that "the snapshot-group entry point was not changed, and the incomplete online merge was canceled when the 3-hour deadline was reached." This case alone cannot attribute the slow merge to a particular disk, iSCSI path, or QEMU defect, and it does not guarantee completion after changing the timeout to 24 hours.
• The configuration change in this article is a workaround. A formal long-term resolution must be based on evidence from the underlying block job, storage performance, and snapshot chain. Do not treat an unlimited timeout extension as a fix.