Source and classification

• Source: Internal Jira/TIC.
• Classification: Product issue.
• Confirmation basis: The source-side logs in the TIC show multiple virtual machines reporting Cannot recv data: Connection reset by peer at the same time. The virtual machines were then confirmed to be on the destination host without having been restarted. The code fix for ZSTAC-77772 always calls domain.abortJob() when migration is canceled, in addition to canceling the block job.
• Internal tracking: TIC-4690 and ZSTAC-77772.

Issue metadata

• Issue ID: TIC-4690
• Component/product: ZVF ZSphere
• Status: The migration cancellation defect has been fixed; the original cause of the libvirt connection interruption remains unknown
• Reported/affected version: 4.8.12 (ZSTAC-77772)
• Last updated: September 3, 2026

Problem description

During live migration of a SharedBlock virtual machine, the libvirt interface might return Cannot recv data: Connection reset by peer. The cancellation path in the old KVM Agent handled only the block job and did not ensure that the domain migration job was canceled. After the control plane receives the failure, it still considers the virtual machine to be on the source host, even though libvirt has actually switched the virtual machine to the destination host. As a result, both the source and destination might retain disk locks.

The available evidence confirms only that incomplete cancellation after the error is the root cause of the residual state and locks. The initial cause of the peer reset remains unknown. Migrating 5 virtual machines simultaneously, high traffic on the destination NIC, and network jitter at the site are only possible triggers and must not be described as confirmed root causes.

Symptoms

• Multiple virtual machines migrate to the same destination host at the same time, and the source KVM Agent reports unable to migrate vm and Cannot recv data: Connection reset by peer.
• The API/control plane reports migration failure and retains ownership on the source host, but the only running virtual machine QEMU/domain is already on the destination host. The virtual machine has not been restarted.
• Subsequent upload, backup, startup, or storage operations might report that the LV is locked by another host.
• If running QEMU processes actually exist on both the source and destination, this is a higher-risk active-active scenario and must not be handled as an ordinary residual-lock issue.

Impact

Control-plane ownership, the actual virtual machine location, and the SharedBlock lock owner become inconsistent. Subsequent migration, backup, expansion, or startup operations might fail. Stopping the virtual machine on the destination by mistake or forcibly unlocking an LV while a process still has it open might cause a service interruption or data corruption.

Workaround

• Immediately pause migration, startup, storage, snapshot, and backup operations for the virtual machine. On the source and destination hosts, use read-only checks to compare the libvirt domain, QEMU command line, and virtual machine UUID. Save the timeline of migration API and Agent errors from the management node. Only one running instance is expected across the entire environment.
• Check the install path, SharedBlock lock owner, and device actually opened by QEMU for every root disk and data disk of the virtual machine. The safe recovery prerequisites in this article are satisfied only when the destination instance is running, there is no QEMU process or open file on the source, and only a residual source-side lock remains.
• There is no verified general temporary method that permits users to deactivate the LV or release sanlock directly. Stop if any virtual machine instance, device opener, or lock owner cannot be uniquely identified.
• Support-assisted method: ZStack Technical Support first corrects the control-plane host ownership to the destination host where the only actual running instance resides. Then, release the residual source activation one disk at a time only for disks where there is no QEMU process or file handle on the source and the destination is the only running instance. Before and after processing each disk, save the lock owner, LV activation node, and destination-side I/O status. Stop immediately and recover from a storage backup if destination-side I/O becomes abnormal, an opener remains on the source, or the lock reappears automatically. Process only one virtual machine at a time.
• Recovery success criteria: Platform ownership matches the only instance on the destination, the source no longer holds the lock, destination-side virtual machine I/O is normal, and a read-only disk/status check succeeds. If processes exist on both sides, deactivation affects destination I/O, the lock reappears automatically, or the virtual machine state changes, stop immediately and do not perform a second forced operation.

Affected products

• ZVF ZSphere 4.8.12 (ZSTAC-77772 Issue Version; confirm other versions by Agent package).
• Deployment and dependent components: SharedBlock data storage, libvirt live migration, and the corresponding KVM Agent version. This article applies only when all listed conditions are met: the destination is the only running instance, no source-side process exists, and the source still holds a lock.

Fix status

• Current status: The migration cancellation logic has been fixed.
• Fixed versions: Regression testing passed on 5.4.0.866. The fix was also merged into the branches corresponding to 4.8.36, ZSphere 4.10.18, and 4.10.20. A branch merge does not mean that every build with the same version contains the fix; confirm the delivered package by Agent commit.
• Related defect/fix record: ZSTAC-77772; zstack-utility commit c92a0ce285876517c9ad131d60a20fa280cb2716, among others. The current fix first attempts to cancel the block job when storage migration is involved and then calls domain.abortJob() regardless of whether storage migration is involved. Cancellation exceptions are logged without masking the original migration error.

Related information

• Related TIC/Jira: TIC-4690 and ZSTAC-77772.
• Validation boundary: Jira once recorded automation coverage, but it was later withdrawn because the test environment could not trigger the defect conditions. This article uses only the manual regression conclusion for 5.4.0.866 and does not count the withdrawn case as valid automation evidence.