Issue/Introduction

This article applies to image upload, template sealing, creating a VM instance from an image, querying an image list, and reconnecting image storage. These issues are generally related to the image source, template sealing procedure, guest operating system initialization tools, image storage status, database permissions, or image backing files.

Symptoms

• After a Windows image exported from VMware Workstation is uploaded, the VM instance created from it cannot enter the operating system after GuestTools is installed and the VM instance is restarted.
• After a VM instance is created from a sealed template, its IP address, gateway, or network configuration is restored to the old value and its network is unreachable.
• The image list API call fails, and the UI also cannot return the image list normally.
• Image storage reconnection fails, or creating a VM instance from an image reports an error.
• The status of image storage is inconsistent with that of primary storage, the image cache, or image backing files.

Environment

• Product: ZCF
• Component: Cloud
• Affected objects: ZStack Cloud; image, template, image storage, ImageStore, Cloudbase-Init, GuestTools, and database permissions
• Version: 4.x-4.8.38

Possible causes

• Cloudbase-Init, VMware Tools, a third-party driver, or an initialization script is installed in the source image and conflicts with the ZStack GuestTools.
• Before template sealing, the system network configuration, cloud-init configuration, udev rules, hostname, or SSH key was not cleaned, causing a new VM instance created from the template to inherit the old configuration.
• Image storage is unavailable or cannot reconnect, the image cache is corrupted, or ImageStore metadata is abnormal.
• ZStack database user permissions are abnormal, causing image list queries and image storage operations to fail.
• The image source contains internal snapshots or a backing chain, or its format does not meet current platform expectations.

Resolution

1. Identify the image source and failure stage

• Record the image UUID, format, source, upload method, image storage type, and zone.
• Determine whether the issue occurs while uploading the image, creating the VM instance, sealing the template, installing GuestTools, querying the image list, or reconnecting image storage.
• If both the UI and API image list queries are abnormal, first check management logs, database permissions, and image storage status.

2. Check template sealing and guest operating system initialization tools

• For a Windows image, focus on Cloudbase-Init, GuestTools, virtio and NIC drivers, SID, hostname, and automatic initialization scripts.
• For a Linux image, focus on cloud-init, NetworkManager, udev NIC rules, /etc/fstab, SSH keys, hostname, and DHCP configuration.
• For an image imported from VMware, first confirm that VMware Tools, legacy-platform drivers, and initialization tools were uninstalled or disabled according to the migration standard.

3. Check image storage and image list queries

zstack-ctl status
grep -a -n "image\|backup storage\|ImageStore\|permission" /usr/local/zstack/apache-tomcat/logs/management-server.log*
zstack-cli
QueryImage uuid=<image-uuid>
QueryBackupStorage uuid=<backup-storage-uuid>

• If neither the UI nor the API can return the image list, confirm whether image storage is unavailable, database permissions are abnormal, or the management service is abnormal.
• If reconnecting image storage fails, continue checking its address, mount path, network connectivity, service process, and storage capacity.

4. Check the image file and backing relationship

qemu-img info <image-file>
qemu-img info -backing-chain <image-file>
qemu-img check <image-file>

• If an image contains a backing chain or internal snapshot, or has an abnormal format, it is not recommended that you use it directly as a production template.
• For a backing-chain image or an image created from a VM instance, continue checking the snapshot chain, ImageStore metadata, and image cache.

5. Verify the result

• Create a test VM instance from the repaired image and verify startup, NICs, volumes, password injection, monitoring, and workload access.
• For a template image, it is recommended that you verify shutdown, restart, migration, cloning, and network reconfiguration at least once.
• If the issue is related to database permissions or image storage status, verify both UI and API queries after the repair.

Additional troubleshooting methods

Basic troubleshooting commands

# Management node services and basic cluster status
zstack-ctl status
systemctl status zstack-ha.service 2>/dev/null || true
systemctl status mariadb.service mysqld.service 2>/dev/null || true
zsha2 status 2>/dev/null || true

# Main management node log and recent errors
ls -lh /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -iE 'ERROR|WARN|Exception|failed|timeout' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

# Basic resources and system status
df -hT
free -m
top -b -n 1 | head -n 30
journalctl -p warning --since '2 hours ago' --no-pager | tail -n 200

Temporary workarounds

• First distinguish a management plane display issue, resource status issue, and underlying workload issue. Consider reconnecting, refreshing caches, retrying tasks, or restarting the management service only after confirming that the underlying status is normal.
• If the management node service must be restarted, first confirm the HA status, current tasks, and workload window. Prefer processing one node at a time and preserve /usr/local/zstack/apache-tomcat/logs/management-server.log*.
• Before repairing a database, forcibly deleting a resource, or reconnecting primary storage or image storage, complete a database backup and record a rollback plan.

Backup, migration, and image checks

zstack-cli QueryLongJob state!=Succeeded
zstack-cli QueryTaskProgress 2>/dev/null || true
zstack-cli QueryBackupStorage
zstack-cli QueryImage name~=<keyword>
grep -iE 'backup|cdp|image|migration|snapshot|qemu-img|nbd|ERROR|Exception|timeout' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 300
ps -ef | egrep 'qemu-img|qemu-nbd|rsync|backup|zstack' | grep -v grep
df -hT

Temporary guidance for backup and migration issues

• When a backup, CDP, migration, or image task fails, first confirm whether the task is still running and whether a background qemu-img or rsync process exists. Avoid initiating the operation again and amplifying concurrency.
• If image storage or backup storage is abnormal, first check capacity, network, mounts, and the task queue. During workload peak hours, prioritize pausing unnecessary backups or reducing concurrency.
• If a snapshot chain, backup chain, or migration path is abnormal, preserve the chain information and logs first. Do not directly delete intermediate files or forcibly clean task records.

Additional information

• Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
• Before a high-risk operation such as a database change, forced deletion, OSD or MON rebuild, pool deletion, service restart, or network change, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.