VM startup fails with "mediated device not found" after vGPU repartitioning

Environment

• Product: ZCF
• Version: 4.x and 5.x (KVM hosts with physical GPU / vGPU support)
• Component: Zstack Cloud
• Trigger/Condition: a VM that was previously bound to a vGPU fails to start after the underlying physical GPU has been re-partitioned (vgpu re-cut), and the vGPU UUID is no longer present on the host.

Symptoms

• VM start task fails.
• A traceback appears in the management node log similar to:

failed to start vm[uuid:cb1334fe78f6408693b5ac3fac733b49 name:440-4] on kvm host[uuid:f8333e4b8c1c4211a3982fbbbacb45b5, ip:xxx.xxx.33.102], because Traceback (most recent call last):
  File "/var/lib/zstack/virtualenv/kvm/lib/python2.7/site-packages/kvmagent/plugins/vm_plugin.py", line 5583, in start_vm
    self._start_vm(cmd)
  File "/var/lib/zstack/virtualenv/kvm/lib/python2.7/site-packages/kvmagent/plugins/vm_plugin.py", line 5499, in _start_vm
    'unable to start vm[uuid:%s, name:%s], libvirt error: %s' % (cmd.vmInstanceUuid, cmd.vmName, str(e)))
KvmError: unable to start vm[uuid:cb1334fe78f6408693b5ac3fac733b49, name:440-4], libvirt error: device not found: mediated device '23cac439-232a-406c-8525-43ee08f55e36' not found

• Inside the host, the referenced mediated device UUID 23cac439-232a-406c-8525-43ee08f55e36 is no longer listed.

Diagnosis

1. Inspect the failing task in the management node UI.
2. On the host, list the available mediated devices (mdev types) for the physical GPU and confirm that the UUID reported in the error is no longer present.
3. Confirm that the vGPU UUID changed because of a recent GPU re-partition operation.

Risks and rollback

• Risk: detaching the vGPU and reattaching a different vGPU may change the device identity presented to the guest. GPU-passthrough workloads that rely on a stable device ID may need their licensing re-bound.
• Rollback: stop the VM, reattach the original vGPU (if it has been recovered), and restart the VM.

Resolution

1. Pre-flight Checks

• Identify the affected VM UUID and the broken vGPU UUID from the error message (23cac439-232a-406c-8525-43ee08f55e36 in the example).
• Confirm that a healthy vGPU with the required profile is available on the same host.

2. Recovery Steps

1. From the management UI, detach (unload) the broken vGPU 23cac439-232a-406c-8525-43ee08f55e36 from the VM.
2. Attach a healthy vGPU of the required profile to the VM.
3. Start the VM. The VM should start successfully using the new vGPU binding.

Verification

• The VM reaches the Running state.
• Inside the guest, the GPU device is visible and the workload that requires it can run normally.

Cause

The VM still references a vGPU UUID that no longer exists on the host. This typically happens after the physical GPU has been re-partitioned (for example, the vGPU was re-cut to a different profile or size), and the VM's binding was not refreshed.

Additional note

• The original case notes that primary storage usage was over 90% at the time; expand primary storage capacity proactively to avoid storage pressure compounding start failures.