Issue/Introduction

This article troubleshoots errors such as block jobs not found and expect N block jobs, only found M during multi-disk backup, and distinguishes storage I/O fluctuation from persistent disk-path inconsistency.

Symptoms

• zstore.log or backup task details expect N block jobs but find only M.
• Some tasks succeed after retry, or multiple virtual machines fail during the same period.
• Another task type consistently reports that a volume installPath cannot be found on the VM; retrying does not help.

Environment

• Product: ZVF ZSphere
• Version: 4.10.16 (direct evidence from the diagnostic branch and running package)
• Components: ImageStore VM backup, zstcli/zstack-store, and KVM/libvirt block jobs
• Deployment: One or more disks to back up; this article covers both temporary block-job count mismatch and a volume absent from current domain XML

Possible causes

1. Slow underlying backup-server I/O or network/libvirt fluctuation makes a block job temporarily invisible during the 3-second polling window.
2. High backup concurrency or read/write rates amplify state desynchronization.
3. Management-node VolumeVO.installPath differs from current libvirt domain XML. This persistent issue is not a temporary missing block job.

Resolution

1. Branch by error text first

1. In the ZSphere UI, open failed task details and record apiId, failure time, VM UUID, and disk UUID. On the management node, paste the actual apiId into the variable:

   KB_API_UUID='<apiId-from-task-details>'
   cd /usr/local/zstack/apache-tomcat/logs || exit 1
   ls -lh management-server.log*
   zgrep -a -n -C 3 "$KB_API_UUID" management-server.log*
   zgrep -a -n "$KB_API_UUID" management-server.log* | grep -E 'json POST|http response'

json POST is the management-node request and http response is the Agent response. Record request URL, target IP, taskuuid/taskUuid, and full response. /vm/volume/takebackup usually targets the VM's ZSphere host, while /imagestore/exportnbdimage usually targets the ImageStore backup server. If absent, verify management node, UUID, and failure date, then search .gz archives; do not proceed from only the last UI error.

2. On the identified ImageStore backup server, query current and rotated zstore logs by taskuuid:

   KB_TASK_UUID='<taskuuid-from-management-server-log>'
   cd /var/log/zstack/zstack-store || exit 1
   ls -lh zstore*
   zgrep -a -n -C 5 "$KB_TASK_UUID" zstore*
   zgrep -a -n "$KB_TASK_UUID" zstore* | grep -iE 'block jobs|only found|not found|iohung|error|fail|timeout'

3. On the identified ZSphere host, query KVM Agent logs with the taskuuid from the request to that host. Do not use the different ImageStore taskuuid:

   KB_TASK_UUID='<taskuuid-from-management-server-log>'
   cd /var/log/zstack || exit 1
   ls -lh zstack-kvmagent.log*
   zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*

4. Record the full error, VM UUID, expected/actual block-job count, and failure time across all three endpoints. Use vim -R LOG_FILE, search with /, use n, and quit with :q!.

• For expect N block jobs, only found M or block jobs not found, go to step 2.
• For unable to find volume[installPath:...] on vm[...], go to step 3. The request disk path does not match current domain XML, so waiting for I/O recovery usually does not help.

2. Observe whether the block job is only temporarily invisible

• Location: The affected VM's current ZSphere host.
• Check: When backup starts or the error reproduces, repeatedly run this read-only command and retain timestamped outputs:

  KB_VM_UUID='<vm-uuid-from-failed-task>'
  virsh qemu-monitor-command "$KB_VM_UUID" '{"execute":"query-block-jobs","arguments":{}}' --pretty

• Expected result: Every disk has a block job, count equals N, and progress fields such as offset change.
• Branch A: If a query is empty but the job reappears and progresses, treat it as temporary visibility/I/O fluctuation and check host-to-backup-server network plus NFS/multipath latency.
• Branch B: If every query lacks the same disk, use step 3 to check whether it remains attached.
• Code boundary: Reviewed image-store/backup/backup.go queries about every 3 seconds and errors on count mismatch. This explains the error form but does not prove slow storage in every case.

3. Compare the logged installPath with current VM XML

• Location: The VM's current ZSphere host.

  KB_VM_UUID='<vm-uuid-from-failed-task>'
  virsh dumpxml "$KB_VM_UUID" | grep -A2 '<source'

• Expected result: Every logged installPath appears in its disk <source>, and disk count matches the backup request.
• Exception handling: If a path remains absent, record it, disk UUID, and full XML. Support with read-only database access checks VolumeVO.vmInstanceUuid, installPath, status, and lastOpDate. Do not modify the database or detach/attach without confirmed ownership.

4. Verify binaries and log locations

• VM backup starts from /usr/local/zstack/imagestore/bin/zstcli on the ZSphere host; diagnostics might be in /tmp/vmback/, /var/log/zstack/zstack-kvmagent.log, and /var/log/zstack/zstcli/.
• Also collect /var/log/zstack/zstack-store/zstore.log on ImageStore and /usr/local/zstack/apache-tomcat/logs/management-server.log on the management node.
• If the ZSphere-host client lacks expected diagnostics, confirm the actual execution host and binary; do not inspect only management-node or backup-storage package sources.

5. Temporary handling and verification

Workaround

• Applicability: Current XML includes every backup disk, missing jobs reappear in repeated queries, or NFS/multipath latency aligns with count mismatch.
• Temporary action: Record the schedule and concurrency, pause batch plans on the backup server, wait for active tasks to end normally, then back up one VM at a time while saving block-job, zstore, and latency samples. Do not kill zstcli/QEMU.
• Branch handling: If low concurrency works, retain staggering, repair confirmed network/NFS/multipath latency, then restore concurrency gradually. If the same job remains absent, stop retrying and follow the path-inconsistency/disk-failure branch.
• Path inconsistency: No verified general self-service method exists. Submit VM/disk UUIDs, missing installPath, domain XML, and read-only database results for ZStack Support to restore platform attachment before backup.
• Do not change ZS_MONITOR_INTERVAL; this backup path does not read it. Do not force-stop block jobs or rewrite VM/disk state.
• Success means every disk has a block job, backup completes, and zstore.log has no count mismatch or missing disk path.
• Rollback: Low concurrency changes only scheduling. If validation fails, keep the original plan configuration but leave it paused and escalate. Restore the recorded schedule/concurrency only after root-cause repair and single-VM success.

Additional information

• For a version-specific fix package, provide actual running binary information from both ZSphere and ImageStore hosts, not only management-node package-source files.
• Block-job tolerance windows and fix status vary by version. Confirm the site build and repeated samples before retrying or changing concurrency.