Issue/Introduction

This article applies to GPU passthrough, vGPU partitioning and restoration, GPU driver installation, NVIDIA vGPU licenses, VM instance startup failures, and abnormal GPU resource display. Confirm the BIOS and IOMMU, host kernel parameters, vfio driver, NVIDIA driver, mdev data, license server, and guest operating system driver together.

Symptoms

• After a GPU is passed through to a VM instance, the guest operating system recognizes it abnormally or driver installation fails.
• Windows Device Manager reports This device cannot find enough free resources that it can use, with error code 12.
• A host displays a vGPU that does not actually exist, and a VM instance fails to start after being scheduled to that host.
• vGPU restoration fails, and two sets of partitioning information or old mdev remnants exist.
• An unlicensed NVIDIA vGPU has a frame-rate limit, unavailable CUDA, or restricted resource allocation.
• After the host GPU driver is upgraded, vGPU partitioning fails or residual records remain in the database.

Environment

• Product: ZCF
• Component: Cloud
• Affected objects: ZStack Cloud; GPU passthrough, vGPU, NVIDIA driver, vfio, IOMMU, mdev, and license server
• Version: 4.x-4.8.38

Possible causes

• Intel VT-d or AMD-Vi is not enabled in the BIOS, or IOMMU is not enabled in the host kernel.
• The vfio or vfio-pci module is not loaded, or a device in the same IOMMU group is still bound to its native driver.
• The NVIDIA host and guest driver versions, guest operating system version, or GRID driver do not match.
• When a GPU with large video memory is passed through, the UEFI or OVMF MMIO space is insufficient.
• During vGPU partitioning or restoration, the vGPU was not detached from the VM instance first, leaving residual mdev data.
• The NVIDIA vGPU license is not configured, CLS or DLS is unreachable, or the gridd service did not load the token.
• Records such as MdevDeviceVO and PciDeviceMdevSpecRefVO in the platform database do not match the actual host status.

Resolution

1. Check basic host capabilities

cat /proc/cmdline
dmesg | grep -i iommu
lsmod | grep vfio
lspci | grep -i nvidia
lspci -v -s <pci-address> | grep -i "Kernel"

• VT-d or AMD-Vi must be enabled in the BIOS.
• IOMMU must be enabled in the kernel parameters, and native drivers such as nouveau and nvidiafb must be blocked as required.
• Before passthrough, confirm that the target GPU and devices in the same IOMMU group meet the vfio binding requirements.

2. Check VM instance passthrough and the guest operating system driver

• In management node logs, focus on hot_plug_pci_device, handle_vfio_irq_conflict, and hot_unplug_pci_device.
• Search KVM agent, libvirt, and guest operating system logs for vfio, pci, nvidia, and iommu.
• For a Windows driver issue, also check Device Manager, system event logs, and the blue-screen minidump.
• For a Linux driver issue, check the DKMS build logs and kernel version compatibility.

3. Handle error 12 for a GPU with large video memory

• For an A40 or another GPU with large video memory, if Windows reports error 12, you can assess the use of UEFI and adjustment of the OVMF 64-bit MMIO space.
• In the applicable scenario, X-PciMmio64Mb is used to set the MMIO space to twice the video memory. ZStack Technical Support must confirm the exact parameter for the environment before it is changed.

4. Check vGPU partitioning, restoration, and residual data

select * from MdevDeviceVO;
select * from PciDeviceMdevSpecRefVO;

• Before upgrading the driver or restoring vGPU partitioning, you should first shut down the VM instance that uses the vGPU and detach the vGPU.
• Uninstalling the driver without first restoring the partitioning might leave residual database records.
• Before repairing the database, you must back it up and have ZStack Technical Support confirm the SQL statement.

5. Check the NVIDIA license

• An unlicensed vGPU might have a frame-rate limit, restricted resource allocation, or disabled CUDA.
• CLS licensing requires the VM instance to access NVIDIA online services. DLS requires an internal license server.
• The Windows token directory is generally %SystemDrive%\Program Files\NVIDIA Corporation\vGPU Licensing\ClientConfigToken.
• The Linux token directory is generally /etc/nvidia/ClientConfigToken; also check the nvidia-gridd service.

Additional troubleshooting methods

Basic troubleshooting commands

# Management node services and basic cluster status
zstack-ctl status
systemctl status zstack-ha.service 2>/dev/null || true
systemctl status mariadb.service mysqld.service 2>/dev/null || true
zsha2 status 2>/dev/null || true

# Main management node log and recent errors
ls -lh /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -iE 'ERROR|WARN|Exception|failed|timeout' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

# Basic resources and system status
df -hT
free -m
top -b -n 1 | head -n 30
journalctl -p warning --since '2 hours ago' --no-pager | tail -n 200

Temporary workarounds

• First distinguish a management plane display issue, resource status issue, and underlying workload issue. Consider reconnecting, refreshing caches, retrying tasks, or restarting the management service only after confirming that the underlying status is normal.
• If the management node service must be restarted, first confirm the HA status, current tasks, and workload window. Prefer processing one node at a time and preserve /usr/local/zstack/apache-tomcat/logs/management-server.log*.
• Before repairing a database, forcibly deleting a resource, or reconnecting primary storage or image storage, complete a database backup and record a rollback plan.

Hardware, driver, security, and upgrade checks

uname -a
rpm -qa | egrep 'zstack|qemu|libvirt|kernel|nvidia|vfio|spice|qxl' | sort
lspci -nn
lsmod | egrep 'vfio|kvm|nvidia|mlx|ixgbe|i40e|bnx|megaraid'
dmesg -T | grep -iE 'error|failed|reset|timeout|nmi|mce|iommu|vfio|gpu|driver' | tail -n 200
systemctl list-units --failed

Temporary guidance for hardware and upgrade issues

• For a driver, GPU, peripheral, or upgrade dependency issue, first confirm the kernel, driver, firmware, and ZStack version combination. Do not directly upgrade the kernel or replace a driver before confirming compatibility.
• For an issue after security hardening or vulnerability remediation, first check the firewall, SELinux, system libraries, listening ports, and service startup permissions. If necessary, roll back the hardening item according to the change plan.
• For an upgrade failure, preserve the upgrade logs, version packages, dependency check output, and current service status. Run continue or retry only after confirming that the continuation conditions are met.

Additional information

• Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
• Before a high-risk operation such as a database change, forced deletion, OSD or MON rebuild, pool deletion, service restart, or network change, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.