Source and classification

• Source: Internal Jira/TIC.
• Classification: Product issue.
• Confirmation basis: The confirmed defect is that the old logic declared ioHung when block-job len/offset remained unchanged for about 1 minute, although a customer task could resume after stalling for about 2 minutes.
• Internal tracking: TIC-4663, ZSV-9963, and ZSTAC-78162.

Issue metadata

• Issue ID: TIC-4663
• Component/product: ZVF ZSphere
• Status: Fixed; delivery matrices for individual ZSphere maintenance branches must be confirmed separately
• Reported/affected version: Old ioHung detection logic; the reported environment backed up a large SAN disk to an NFS backup server
• Last updated: September 2, 2026

Problem description

The old logic failed a task as ioHung whenever block-job len/offset did not change during a fixed short window. A real storage path might only pause temporarily and later continue, making this a false positive. The fix logs one "stalled" event and one "recovered" event without terminating the task. For a real I/O hang, the task can still be canceled from the UI and cleaned up by the underlying yank operation.

Symptoms

• zstore/task errors directly return ioHung, but the logs contain no corresponding job-cancel or yank record.
• The QMP block-job offset stops changing temporarily and then increases again, unlike a permanent I/O hang.
• Large backups from SAN disks to an NFS backup server expose long pauses more readily, but the storage type is not the product root cause.

Impact

A recoverable backup is marked failed early, preventing a large virtual machine from obtaining a new backup for a long time. Conversely, disabling all I/O monitoring can leave a real hang occupying task resources indefinitely. Use the official fix, which retains logging and UI cancellation.

Workaround

• There is no verified general self-service parameter workaround. Do not disable monitoring or change the detection time yourself; a fixed threshold cannot distinguish a temporary pause from a real I/O hang.

Confirm logs and progress (read-only)

1. In the ZSphere UI, open failed task details and record apiId, VM UUID, failure time, and ZSphere host. Run on the management node:

   KB_API_UUID='<apiId-from-task-details>'
   cd /usr/local/zstack/apache-tomcat/logs || exit 1
   ls -lh management-server.log*
   zgrep -a -n -C 3 "$KB_API_UUID" management-server.log*
   zgrep -a -n "$KB_API_UUID" management-server.log* | grep -E 'json POST|http response'

Record the Agent IP and taskuuid/taskUuid for requests such as /vm/volume/takebackup and /imagestore/exportnbdimage. If current logs have no result, use the same command to query management-server-DATE-SEQUENCE.log.gz archives; if still absent, first verify the management node and UUID.

2. On the image storage (ImageStore) backup server identified by the request, query current and archived zstore logs by taskuuid:

   KB_TASK_UUID='<taskuuid-from-management-server-log>'
   cd /var/log/zstack/zstack-store || exit 1
   ls -lh zstore*
   zgrep -a -n -C 5 "$KB_TASK_UUID" zstore*
   zgrep -a -n "$KB_TASK_UUID" zstore* | grep -iE 'iohung|offset|block job|error|fail|timeout|yank|cancel'

3. On the VM's ZSphere host, use the taskuuid from the request sent to that host to query KVM Agent and use the VM UUID to inspect the current block job. Do not use the different taskuuid sent to ImageStore:

   KB_TASK_UUID='<taskuuid-from-management-server-log>'
   KB_VM_UUID='<vm-uuid-from-task-details>'
   cd /var/log/zstack || exit 1
   zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*
   virsh qemu-monitor-command "$KB_VM_UUID" '{"execute":"query-block-jobs","arguments":{}}' --pretty

While the task runs, manually repeat the last read-only command every 30 seconds and save at least 3 outputs. An increasing offset indicates that old logic might have mistaken a pause for ioHung. If the block job remains unchanged while NFS/SAN continues reporting I/O errors, address the actual storage fault first. The 30-second interval is an evidence-sampling interval, not a product threshold. An empty QMP list after task completion does not prove that no block job existed; use logs from the failure window.

4. To browse current logs, use vim -R zstore.log or vim -R /var/log/zstack/zstack-kvmagent.log, enter /, paste the UUID, press Enter, use n for the next match, and enter :q! to quit.

• Support-assisted method: Use a version containing the fix or a zstack-store package matching the exact branch. Save the original package/checksum and logs before replacement and verify one large-disk backup. If the new logs still fail on a brief pause, restore the original package and verify the branch.
• For a persistent real hang, cancel the task in the platform UI and confirm that yank eventually occurs and QMP has no residue. Do not kill zstore/QEMU directly or delete the backup target.
• Verification criteria: Backup of the same large virtual machine completes; a brief pause produces only stalled/recovered logs rather than immediate failure; manually canceling a real hang ends the task and leaves no QMP block job.

Affected products

• ZVF ZSphere: 4.10.18 is the direct reported/affected evidence point. Versions 5.1.0.388 and 5.4.0.1087 are only fix-simulation validation records; see Fix status.

Fix status

• Current status: The product root cause and fix direction are confirmed.
• Fixed versions: ZSTAC-78162 records 5.4.0 and 4.8.36; ZSV-9942 was verified on 4.10.18; ZSV-12556 was verified on 5.1.0.388. One simulation on 4.10.20 is not equivalent to a site hang and cannot alone serve as complete acceptance.
• Related defect/fix records: ZSTAC-78162, ZSV-9942, ZSV-9963, and ZSV-12556

Related information

• Related TIC/Jira: TIC-4663, ZSV-9963, ZSV-12556, and ZSTAC-78162.