Issue

Use this article to troubleshoot GPU passthrough task timeouts.

Environment

• Product: ZCF
• Component: Cloud
• Affected object: GPU passthrough
• Versions: 4.x, 5.x
• Scenario: GPU passthrough task timeout

Symptoms

• A VM fails to start, a task times out, or qemu reports an error after GPU/vGPU passthrough or loading.

Possible causes

• IOMMU, vfio binding, device groups, drivers, GPU model, license, or physical-host hardware compatibility is abnormal.

Resolution

1. Confirm the GPU model, driver, IOMMU/vfio state, and VM XML.
2. Check passthrough-task, kvmagent, and libvirt/qemu logs.
3. Cross-validate with physical hosts of the same model to distinguish hardware, driver, and platform logic.

Key command

lspci | grep -i -E "nvidia|vga|3d"
dmesg | grep -i -E "vfio|iommu|nvidia|pcie"
grep -a -nE "vfio|iommu|gpu|nvidia|qemu|libvirt" /usr/local/zstack/apache-tomcat/logs/management-server.log* /var/log/libvirt/qemu/*.log 2>/dev/null

Remediation

• Based on the troubleshooting result, repair the affected component, configuration, data state, or connection path.
• If databases, underlying storage, snapshot chains, VM disks, GPU passthrough, or management-service restarts are involved, first confirm backups, scope of impact, and rollback paths.

Verification

• Confirm that the symptom described in this article no longer occurs.
• UI, API, management-node logs, and underlying command output corroborate one another.
• Observe at least one related task or alarm cycle to confirm that the issue does not recur.

Risks and rollback

• Before remediation in production, confirm customer service impact, the change window, and the rollback plan.
• For data deletion, database modifications, snapshot-chain repairs, underlying file operations, or hardware operations, ZStack recommends contacting technical support for confirmation.

Notes

• For VM startup, migration, HA, Unknown status, or forced stop operations, first confirm the actual VM running location to prevent split-brain operation, dual writes, or unintended service interruption.
• For host drivers, GPUs, RAID, NICs, IPMI/BMC, firmware, or BIOS changes, consider hardware compatibility and vendor recommendations, and prepare a rollback plan in advance.
• If database modifications, underlying file operations, lock release, service restarts, driver or firmware upgrades, or bulk resource changes are required, contact ZStack technical support before proceeding.