Issue/Introduction

This article troubleshoots subsequent Veeam backup failures after a disable CBT operation times out.

Symptoms

• The first backup waits about 30 minutes at disable CBT and then fails.
• The old CBT task remains in the Running state in ZSphere.
• The next backup finds the old task and then fails because the old CBT volume cannot be found. Repeated retries for the same virtual machine continue to fail.
• api-proxy logs might also show enable CBT or export NBD timeouts.

Environment

• Product: ZVF ZSphere
• Version: 5.1.0
• Components: ZSphere, api-proxy, Veeam CBT, and temporary CBT volumes
• Deployment: ZSphere 5.1.0 with api-proxy 1.0.4, where Veeam invokes CBT backup

Possible causes

• Deleting a temporary volume times out on the control plane. The data plane might have deleted it, while the task or reference record was not cleaned up. This still requires reproduction in the customer environment.
• Excessive backup concurrency expands the timing window for CBT creation, export, and cleanup.
• External CBT and platform backup run at the same time, causing a separate mutual-exclusion issue.

Resolution

1. Reconstruct the task chain from the first timeout to subsequent failure

• Check: The latest successful backup, first failed task, and next failed Veeam/CBT task for the same virtual machine.
• Location: Record the VM UUID, Veeam job, CBT task UUID, and exact time, then query as follows:

1. Open the first failed task in the ZSphere UI and record its apiId. Run on the management node:

     KB_API_UUID='<apiId-from-task-details>'
     cd /usr/local/zstack/apache-tomcat/logs || exit 1
     ls -lh management-server.log*
     zgrep -a -n -C 3 "$KB_API_UUID" management-server.log*
     zgrep -a -n "$KB_API_UUID" management-server.log* | grep -iE 'json POST|http response|cbt|nbd|delete|timeout'

Record the destination Agent IP, URI, taskuuid/taskUuid, and time from the request and response. If the first timeout and subsequent failure have different apiId values, query both; do not retain only the later error.

2. On the ZSphere host addressed by the request, query current and rotated logs by taskuuid:

     KB_TASK_UUID='<taskuuid-from-management-server-log>'
     cd /var/log/zstack || exit 1
     ls -lh zstack-kvmagent.log*
     zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*
     zgrep -a -n "$KB_TASK_UUID" zstack-kvmagent.log* | grep -iE 'cbt|nbd|volume|delete|not found|error|fail|timeout'

3. Collect logs from at least 5 minutes before disable CBT starts through 5 minutes after timeout. A single wait is about 30 minutes in this scenario, so capturing only the error instant omits the causal chain among creation, deletion, and timeout. To inspect current logs, use vim -R LOG_FILE, enter /, paste the UUID, press Enter, use n for the next match, and enter :q! to quit.

• Expected result: Map every failure to a specific stage such as enable CBT, export NBD, disable CBT, or stop VM, rather than retaining only the final Veeam error.
• Exception handling: Do not clean up an old task if logs do not cover the failure or resource UUIDs are incomplete. Collect a complete window at the next reproduction first.

2. Compare the control-plane CBT task with the data-plane temporary volume

• Check: Old CBT task state/update time, related temporary-volume reference, and whether the volume actually exists on the host.
• Location: Query the old task in the UI/CLI and search logs for disable cbt, cbt volume, delete, and not found. Support staff perform a read-only volume check on the destination host.
• Expected result: A task in the Running state has its temporary volume, or a completed task and all its references are cleaned up.
• Exception handling: A task that remains in the Running state after the temporary volume was deleted indicates control-plane/data-plane inconsistency. Do not delete database records directly or guess an LV name for cleanup.

3. Exclude port conflicts and concurrency amplification

• Check: 16443 errors, enable CBT task timeout after 10800 seconds, export NBD timeout in api-proxy logs, and backup concurrency during the same period.
• Location: The api-proxy log directory may vary by deployment. Find the actual files read-only before searching current and archived logs:

  find /var/log/zstack /usr/local/zstack -type f \( -iname '*api-proxy*.log*' -o -iname '*apiproxy*.log*' \) -print 2>/dev/null
  find /var/log/zstack /usr/local/zstack -type f \( -iname '*api-proxy*.log*' -o -iname '*apiproxy*.log*' \) -print0 2>/dev/null | xargs -0 -r zgrep -a -n -iE '16443|enable CBT task timeout|export.*nbd|disable.*cbt'

If the first command finds nothing, confirm the path from the log bundle on the Veeam/api-proxy deployment node; do not assume the logs are on the management node. Count virtual machines in the same job and overlapping jobs by minute, and determine whether platform backup simultaneously operates on the same VM.

• Expected result: The api-proxy port works, Veeam and platform backup do not concurrently operate on the same VM, and low-concurrency windows no longer produce many CBT/NBD timeouts.
• Exception handling: Resolve an api-proxy listener conflict first if the port is occupied. Converge on residual state as the main problem only if old-task/volume-missing errors remain reproducible at low concurrency.

4. Stagger tasks to limit impact, then have support repair residual state

Workaround

• Verified self-service method: The recorded site recommendation was to temporarily limit each Veeam backup job to no more than 5 virtual machines and stagger different jobs. This is a case-specific mitigation value, not a general product limit. Pause automatic retries for the affected VM and observe one low-concurrency run.
• Support-assisted method: Low concurrency cannot repair an existing stale CBT state. After confirming the task UUID, actual temporary-volume state, database references, and maintenance window, ZStack Technical Support provides a cleanup method for the exact version. This article provides no general command for directly modifying the database or deleting an LV.
• Expected result: The old CBT task no longer remains in the Running state, and task references match temporary volumes. One complete enable, export, and disable flow succeeds in a non-concurrent window, and later backups no longer report that the old CBT volume cannot be found.
• Exception handling: If many 10800 seconds or export NBD timeouts still occur at low concurrency, preserve api-proxy and host logs and proceed to performance/path analysis. Do not continue cleaning up state.

Additional information

• Support information must include the exact time window, resource and task UUIDs, complete error text, and relevant component logs.
• Do not extrapolate one site result to all versions. If the root cause remains open or an operation might affect services or data, stop self-service handling and submit the evidence listed in this article.