Issue/Introduction

This article troubleshoots large or multi-disk backups that run for a long time and fail with expect N block jobs, only found M when the ImageStore backup directory is on NFS. The error means progress monitoring expected N libvirt block jobs but found M; it alone does not prove NFS timeout is the only cause.

Symptoms

• Small full backups complete, while large ones fail after running for a long time.
• Task details or zstore show:

  expect 2 block jobs, only found 1

• The error chain might include APICreateVmBackupMsg, zstcli ... batbak, and CreateVmBackup.
• ImageStore might briefly disconnect; establish sequence from management-node, ZSphere-host, and ImageStore logs.

Environment

• Product: ZVF ZSphere
• Version: 4.10.16.405 (site build from source log-bundle filename; do not extrapolate)
• Components: VM backup, ImageStore backup server, NFS, zstcli/zstack-store, and KVM/libvirt block jobs
• Deployment: ImageStore backup directory mounted over NFS; the recorded case was a large 2-disk full backup

Possible causes

This symptom can be caused by one or more of the following:

1. NFS server performance, client path, or mount instability causes sustained latency/retransmission and one block job fails first.
2. Large, multi-disk, or concurrent backups amplify I/O pressure across KVM, ImageStore, NFS, and third-party backup software.
3. A disk never obtains a block job, its job exits early, or it is no longer visible when queried.
4. The disk set in the management-node request differs from current VM domain XML.

The source converged on storage and third-party software but did not prove one failure point. Changing ZS_STORAGE_HEARTBEAT_TIMEOUT and replacing a runtime package did not reliably fix the 5 TB failure, so neither is a general solution.

Resolution

1. Record search criteria from task details

1. Open the failed Create backup task in ZSphere.
2. Record all of the following information instead of saving only the last error line:

• Operation start time, failure time, and time zone.
• apiId.
• VM UUID.
• UUID of every disk to be backed up.
• ImageStore backup server UUID.
• Expected and actual block-job counts, such as expect 2 and only found 1.

3. If only a short error appears, copy complete details or export the operation log.

• Expected result: A unique time window, apiId, VM UUID, and disk UUIDs.
• Exception handling: Without apiId, search by minute-precise time, VM UUID, and backup name; do not substitute another retry's ID.

2. Find request destination and taskUuid on the management node

• Location: The management node that ran the task; query both in a dual-MN environment if necessary.
• Paste the actual values from the task details into the variables:

  KB_API_UUID='<apiId-from-task-details>'
  cd /usr/local/zstack/apache-tomcat/logs || exit 1
  ls -lh management-server.log*
  zgrep -a -n -C 5 "$KB_API_UUID" management-server.log*
  zgrep -a -n "$KB_API_UUID" management-server.log* | grep -E 'json POST|http response|taskuuid|taskUuid|error|fail'

• How to read the results:

1. First locate the API request containing the apiId and the final error.
2. Then locate the json POST in the same time window and record its URL, destination IP, and taskuuid or taskUuid from the request body.
3. The json POST destination IP determines which ZSphere host or ImageStore backup server to access next. Different requests might use different taskUuid values; do not mix them.

• Expected result: Identify KVM/ImageStore requests, destination IPs, IDs, and responses.
• Exception handling: Search rotated and .gz files after verifying date; if absent, verify management node, time zone, and task identity.

3. Query zstore on ImageStore

• Location: The ImageStore backup server identified by the request in step 2.
• Prefer the taskUuid in the request sent to ImageStore. If it does not match, cross-check with the VM UUID and failure minute:

  # Replace the variable values with the actual UUIDs from the failed task
  KB_TASK_UUID='task-uuid-placeholder'
  KB_VM_UUID='vm-uuid-placeholder'
  cd /var/log/zstack/zstack-store || exit 1
  ls -lh zstore*
  zgrep -a -n -C 8 "$KB_TASK_UUID" zstore*
  zgrep -a -n -C 8 "$KB_VM_UUID" zstore* | grep -iE 'batbak|block jobs|only found|not found|iohung|timeout|error|fail'

• Expected result: Find CreateVmBackup/batbak start, progress, end, and the first missing/failing job.
• Exception branches:
• If NFS I/O, mount, or ImageStore process errors precede the block-job count mismatch, go to step 5 and focus on NFS.
• If ImageStore remains online throughout but one block job is missing, continue to step 4 and compare actual KVM jobs with VM disks.
• If the current zstore.log has no record, search rotated logs in the same directory by failure date instead of checking only the current file.

4. Compare block jobs and VM disks on the ZSphere host

• Location: The current ZSphere host of the VM identified by the request in step 2.
• First query the KVM Agent log by the taskUuid in the request sent to that ZSphere host:

  KB_TASK_UUID='<taskUuid-from-request-sent-to-KVM>'
  KB_VM_UUID='<vm-uuid-from-failed-task>'
  cd /var/log/zstack || exit 1
  ls -lh zstack-kvmagent.log*
  zgrep -a -n -C 8 "$KB_TASK_UUID" zstack-kvmagent.log*
  zgrep -a -n -C 5 "$KB_VM_UUID" zstack-kvmagent.log* | grep -iE 'backup|block|qmp|libvirt|error|fail|timeout'

• While the issue is reproducing, repeatedly query the VM's current block jobs. These commands are read-only and do not terminate a job:

  KB_VM_UUID='<vm-uuid-from-failed-task>'
  virsh qemu-monitor-command "$KB_VM_UUID" '{"execute":"query-block-jobs","arguments":{}}' --pretty
  virsh dumpxml "$KB_VM_UUID" | sed -n '/<disk /,/<\/disk>/p'

• Expected result: Every disk to be backed up has a corresponding block job, the number of jobs matches N in the task, progress continues to change, and every disk to be backed up appears in the VM XML.
• Exception branches:
• If the same disk is always missing, record its UUID, VM XML, and logs. Do not modify the database or reattach the disk to force creation of a job.
• If the job existed and then disappeared, search backward for that job's first error instead of treating the final only found M as the root cause.
• If every job exists but progress remains unchanged for a long time, preserve multiple query results and align the NFS performance timeline in step 5.

5. Check NFS mount, capacity, latency, and retransmission

• Location: The ImageStore backup server that actually mounts the NFS backup directory.
• First confirm the actual backup directory from the existing mount list, then enter it in the variable. Do not guess the path:

  findmnt -t nfs,nfs4 -o TARGET,SOURCE,FSTYPE,OPTIONS
  nfsstat -m
  KB_BACKUP_PATH='<nfs-backup-directory-confirmed-in-the-previous-step>'
  findmnt -T "$KB_BACKUP_PATH" -o TARGET,SOURCE,FSTYPE,OPTIONS
  df -hT "$KB_BACKUP_PATH"
  command -v nfsiostat >/dev/null 2>&1 && nfsiostat 1 10 "$KB_BACKUP_PATH"
  command -v iostat >/dev/null 2>&1 && iostat -xz 1 10
  dmesg -T | tail -n 300 | grep -iE 'nfs|rpc|not responding|timed out|timeout|reset'

• Expected result: Correct writable NFS mount, sufficient space, and no sustained aligned latency, retransmission, server-not-responding, timeout, or disk saturation.
• Exception handling:
• If the mount is missing, read-only, or out of space, stop new backups and have the storage administrator restore NFS service and capacity first.
• If latency, retransmission, or server-side disk saturation aligns with the failure, record NFS client output and have the storage administrator provide server network and disk metrics for the same time window.
• If both client and server metrics are normal, do not continue attributing the issue to NFS. Return to step 4 and inspect the specific missing block job, libvirt/qemu version, and third-party backup software logs.

6. Compare with low concurrency

1. Let running tasks end normally; if cancellation is required, use the supported UI and confirm completion. Do not kill zstcli, qemu, or libvirt.
2. Pause batch schedules on the same ImageStore.
3. During stable NFS conditions, run the same backup for one VM at a time with the same VM, disks, and destination.
4. Repeat evidence collection from steps 2 through 5 and compare.

• Interpret the result:
• If low concurrency completes but high concurrency or peak periods fail consistently, the storage, path, or third-party software pressure branch is supported. Continue to the workaround in step 7.
• If low concurrency still fails on the same disk and at the same stage, do not retry repeatedly. Submit the three sets of logs, VM XML, NFS metrics, and third-party software logs to ZStack Technical Support for further investigation.

7. Temporary handling, rollback, and verification

Workaround

• Applicability: Steps 5 and 6 confirm NFS or third-party performance pressure and no backup, recovery, or block job remains active.
• Temporary action: Pause batch schedules, avoid storage peaks, and run one VM backup at a time per ImageStore. This only reduces probability and does not replace storage repair.
• Root-cause handling: Storage administrators repair confirmed NFS capacity, network, server disk, or third-party performance. Plan a separate maintenance window, backup, and rollback before changing mounts, backend, or software.
• Not applicable: Do not bypass low-concurrency failure, disk-set mismatch, persistent missing job, or open cause through retries, ImageStore restart, process kill, database changes, or backup-data deletion.
• Rejected historical method: Do not replace zstack-store.bin or set ZS_STORAGE_HEARTBEAT_TIMEOUT yourself. These did not reliably resolve the 5 TB failure. Any diagnostic package requires exact-build scope, original backup/checksum, and rollback from support.
• Rollback: Scheduling only is changed. If low concurrency fails, stop tasks, retain but keep the original plan paused, and escalate. Restore concurrency only after storage repair validation.
• Verification: The same full backup completes with the same disks/destination, no count mismatch occurs in MN/KVM/zstore logs, and NFS has no aligned sustained exception.

Additional information

• Submit task details, exact time/time zone, apiId, VM/disk/ImageStore UUIDs, MN/KVM/zstore logs, VM XML, repeated block-job queries, NFS client/server metrics, and third-party logs.
• only found M is a monitoring result, not necessarily the first fault. Find the earliest I/O, NFS, libvirt/qemu, or third-party error.
• Reviewed zstack-store code queries about every 3 seconds and errors on mismatch; this explains semantics but does not prove a polling-window cause at the 4.10.16.405 site.