Issue/Introduction

This article applies when a VM instance fails to start or enter the operating system, the console displays a black or blue screen, the VM instance pauses abnormally or crashes, data is abnormal after a restart, or internal monitoring has no data. The issue might originate from virtual hardware configuration, system boot, file system corruption, virtio or GuestTools, QEMU logs, storage failures, a platform type mismatch after migration, or behavior inside the guest operating system.

Symptoms

• A VM instance fails to start, displays no booting device found on the console, enters the UEFI shell, or restarts repeatedly.
• A Windows VM instance displays a blue or black screen, starts slowly, or cannot recognize a NIC or volume after migration.
• A VM instance pauses abnormally, times out during recovery, or crashes, and the QEMU logs contain an error such as Resource temporarily unavailable.
• VM instance internal monitoring has no or inaccurate data, or the details page loads slowly.
• The VM instance system or data is abnormal after expansion, restart, migration, or backup.

Environment

• Product: ZCF
• Component: Cloud
• Affected objects: ZStack Cloud; VM instance, QEMU, libvirt, virtio, GuestTools, userdata, internal monitoring, and volume
• Version: 4.x-4.8.38

Possible causes

• The VM instance boot order, BIOS or UEFI mode, platform type, virtio switch, or volume bus type does not match.
• The VM instance file system, LVM or PV, root volume, or boot record is damaged.
• After VMware, V2V, or OVF migration, a driver, platform type, UEFI or BIOS, NIC, or volume device is incompatible.
• A Windows evaluation version, scheduled task in the guest operating system, third-party backup, or workload program triggers a restart or blue screen from inside the system.
• After GuestTools or Cloudbase-Init is installed or configured, the VM instance is not restarted properly, or DHCP or userdata is not enabled on the L3 network.
• A QEMU or libvirt resource, balloon, storage I/O, or underlying host failure causes a crash, pause, or recovery timeout.

Resolution

1. Identify the failure type

• Record the VM instance UUID, name, host, primary storage, failure time, and most recent operation.
• Distinguish startup failure, an internal operating system failure, a console open failure, a post-migration failure, a post-expansion failure, and a monitoring display issue.
• If multiple VM instances on the same host are abnormal at the same time, first check the host, primary storage, QEMU or libvirt, and network.

2. Check the VM instance configuration and console symptom

• Confirm whether the boot order prioritizes the volume and whether BIOS or UEFI matches the source system.
• For Windows or an older operating system, check virtio, VirtioSCSI, the platform type, graphics driver, and Cloudbase-Init.
• For Linux, check partitions, LVM, /etc/fstab, the kernel, initramfs, a read-only file system, and rescue mode availability.

3. Check QEMU, libvirt, and management logs

grep -a -n "<vmUuid-or-error-keyword>" /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -a -n "<vmUuid-or-error-keyword>" /var/log/libvirt/qemu/*.log
journalctl -u libvirtd --since "<start-time>" --until "<end-time>"

• If the QEMU logs report a resource shortage, balloon, storage I/O, migration, watcher, or lock issue, continue troubleshooting the primary storage and host resources together.
• If the management logs report a state lock, task lock, or operation timeout, confirm whether concurrent migration, backup, snapshot, or CDP tasks exist.

4. Check the root volume, volumes, and file system

• If the VM instance cannot enter the system after startup failure or expansion, first create a snapshot or clone and then enter rescue mode to inspect it.
• If LVM or PV is missing, the file system is read-only, or partition expansion is abnormal, preserve the evidence before performing any destructive repair.
• If the VM instance status rolls back or data is abnormal after a storage failure, also inspect primary storage, snapshots, backups, and operation history.

5. Check internal monitoring and GuestTools

• Confirm that DHCP and userdata are enabled on the L3 network and that the VM instance can access 169.254.169.254.
• On Windows, check GuestTools, QGA, Cloudbase-Init, and the forwarding service.
• If a password, NIC, IP address, details-page loading, or monitoring issue occurs after GuestTools is installed, verify GuestTools version and the restart method.

Additional troubleshooting methods

Basic troubleshooting commands

# Management node services and basic cluster status
zstack-ctl status
systemctl status zstack-ha.service 2>/dev/null || true
systemctl status mariadb.service mysqld.service 2>/dev/null || true
zsha2 status 2>/dev/null || true

# Main management node log and recent errors
ls -lh /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -iE 'ERROR|WARN|Exception|failed|timeout' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

# Basic resources and system status
df -hT
free -m
top -b -n 1 | head -n 30
journalctl -p warning --since '2 hours ago' --no-pager | tail -n 200

Temporary workarounds

• First distinguish a management plane display issue, resource status issue, and underlying workload issue. Consider reconnecting, refreshing caches, retrying tasks, or restarting the management service only after confirming that the underlying status is normal.
• If the management node service must be restarted, first confirm the HA status, current tasks, and workload window. Prefer processing one node at a time and preserve /usr/local/zstack/apache-tomcat/logs/management-server.log*.
• Before repairing a database, forcibly deleting a resource, or reconnecting primary storage or image storage, complete a database backup and record a rollback plan.

VM instance and compute node checks

zstack-cli QueryVmInstance uuid=<vm-uuid>
zstack-cli QueryHost uuid=<host-uuid>
zstack-cli QueryVolume vmInstanceUuid=<vm-uuid>
virsh list --all
virsh domstate <vm-name-or-id>
virsh dumpxml <vm-name-or-id> > /tmp/vm.xml
systemctl status libvirtd zstack-kvmagent 2>/dev/null || true
grep -iE 'ERROR|Exception|failed|timeout|qmp|libvirt|domain' /var/log/zstack/zstack-kvmagent.log | tail -n 200
grep -iE '<vm-uuid>|<vm-name>' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

Temporary guidance for VM instance issues

• When a VM instance status is abnormal, compare the management plane, libvirt, QEMU process, and workload status before stopping or deleting based only on the UI status.
• For a black console, first check VNC or SPICE ports, the proxy, browser, host agent, and VM instance graphics configuration. If the workload is normal, you can temporarily use SSH or RDP sign-in instead.
• After a startup failure or abnormal shutdown, preserve the XML, agent logs, and management logs first. If necessary, migrate, reconnect, or refresh the status first. Before a forced operation, confirm volume and snapshot chain integrity.

Additional information

• Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
• Before a high-risk operation such as a database change, forced deletion, OSD or MON rebuild, pool deletion, service restart, or network change, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.