Issue/Introduction

This article applies to abnormal qemu or libvirt binaries or dependencies after an upgrade, abnormal repositories, third-party authentication synchronization failures, management service startup failures caused by licensing, a mismatched UKEY or localized-platform license, and abnormal CPU license recognition. For an upgrade or license issue, check the release branch, ISO or local repository, RPM history, license file, machine fingerprint, service dependencies, and product compatibility boundary together.

Symptoms

• After qemu is upgraded, a binary reports an error, a library function is missing, or retrieving the qemu version is abnormal.
• After an upgrade, a host is disconnected, a dependency package cannot be found, or a repository is abnormal.
• After an upgrade, third-party authentication, LDAP, or SSO synchronization fails and the logs contain SSLHandshakeException.
• After a management node restart, the management service fails to start and reports a license or authorization issue.
• A UKEY for a localized platform cannot be recognized, or authorization cannot start after a dual management node environment is reinstalled.
• A Hygon CPU server reports an abnormal socket count, causing abnormal license consumption.

Environment

• Product: ZCF
• Component: Cloud
• Affected objects: ZStack Cloud; upgrade, qemu, libvirt, repository or Yum, license, UKEY, LDAP or SSO, and management service
• Version: 4.x-4.8.38

Possible causes

• The upgrade ISO, local repository, qemu-kvm-ev repository, or repository file is inconsistent, causing incorrect dependency packages.
• The qemu or libvirt package upgrade is incomplete, and a dynamic library or function symbol does not match the binary.
• The third-party authentication service certificate, domain name or IP address, SAN, or SSL policy does not match the validation logic in the new version.
• The license type, machine fingerprint, number of license files, temporary license expiration, or dual management node synchronization status is abnormal.
• Recognition logic in the current version does not match license counting for specific hardware or CPUs.

Resolution

1. Confirm the upgrade path and version information

zstack-ctl status
yum history
rpm -qa | egrep "qemu|libvirt|zstack|nginx"
cat /etc/yum.repos.d/*.repo

• Record the pre-upgrade and post-upgrade versions, ISO name, patch package, and whether an offline repository or manually installed RPM was used.
• If qemu or libvirt is abnormal, compare the package versions and dynamic library paths with a healthy node.

2. Check the management service and dependencies

zstack-ctl status
systemctl status zstack-ui-nginx --no-pager
journalctl -u zstack-ui-nginx --since "<start-time>"
grep -a -n "ERROR\|license\|SSLHandshake\|qemu" /usr/local/zstack/apache-tomcat/logs/management-server.log*

• If a defense edition or customized edition has a missing dependency, confirm whether the system ISO is compatible and whether Nginx or a dependency package was accidentally deleted.
• If third-party authentication fails, check the certificate SAN, whether access uses an IP address, and whether the certificate validation policy meets the requirements of the current version.

3. Check the license

• Confirm the license type, machine fingerprint, CPU socket recognition, number of temporary license files, and whether an expired license exists.
• After reinstalling a dual management node environment or synchronizing the database, do not directly clear license-related data. First confirm the accesskey and original license retention policy.

4. Define the repair boundary

• For version rollback, patch replacement, license update, dependency package repair, or SSO package rebuild, specify the applicable versions and rollback path.
• After repairing qemu or libvirt, verify VM instance startup, migration, backup, and disaster recovery features.

Additional troubleshooting methods

Basic troubleshooting commands

# Management node services and basic cluster status
zstack-ctl status
systemctl status zstack-ha.service 2>/dev/null || true
systemctl status mariadb.service mysqld.service 2>/dev/null || true
zsha2 status 2>/dev/null || true

# Main management node log and recent errors
ls -lh /usr/local/zstack/apache-tomcat/logs/management-server.log*
grep -iE 'ERROR|WARN|Exception|failed|timeout' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

# Basic resources and system status
df -hT
free -m
top -b -n 1 | head -n 30
journalctl -p warning --since '2 hours ago' --no-pager | tail -n 200

Temporary workarounds

• First distinguish a management plane display issue, resource status issue, and underlying workload issue. Consider reconnecting, refreshing caches, retrying tasks, or restarting the management service only after confirming that the underlying status is normal.
• If the management node service must be restarted, first confirm the HA status, current tasks, and workload window. Prefer processing one node at a time and preserve /usr/local/zstack/apache-tomcat/logs/management-server.log*.
• Before repairing a database, forcibly deleting a resource, or reconnecting primary storage or image storage, complete a database backup and record a rollback plan.

UI, account, permission, and license checks

# Management service, time, and frontend access
zstack-ctl status
date
chronyc sources -v 2>/dev/null || ntpq -p 2>/dev/null || true
curl -k -I https://<management-ip>:5000/ 2>/dev/null || curl -I http://<management-ip>:8080/

# Management log keywords
grep -iE 'login|token|license|permission|account|project|billing|invoice|ERROR|Exception' /usr/local/zstack/apache-tomcat/logs/management-server.log | tail -n 200

Temporary guidance for UI and permission issues

• For a sign-in or page issue, first exclude browser cache, client time drift, management node time synchronization, and reverse proxy or certificate issues.
• For an account, project, permission, billing, or invoice issue, first record the account UUID, project UUID, operation path, and API response. Do not directly change database permission or billing records.
• If license upload fails, first check the product version, license file validity period, machine code, and management node time. You can temporarily restore the original license or pause the upgrade change window.

Hardware, driver, security, and upgrade checks

uname -a
rpm -qa | egrep 'zstack|qemu|libvirt|kernel|nvidia|vfio|spice|qxl' | sort
lspci -nn
lsmod | egrep 'vfio|kvm|nvidia|mlx|ixgbe|i40e|bnx|megaraid'
dmesg -T | grep -iE 'error|failed|reset|timeout|nmi|mce|iommu|vfio|gpu|driver' | tail -n 200
systemctl list-units --failed

Temporary guidance for hardware and upgrade issues

• For a driver, GPU, peripheral, or upgrade dependency issue, first confirm the kernel, driver, firmware, and ZStack version combination. Do not directly upgrade the kernel or replace a driver before confirming compatibility.
• For an issue after security hardening or vulnerability remediation, first check the firewall, SELinux, system libraries, listening ports, and service startup permissions. If necessary, roll back the hardening item according to the change plan.
• For an upgrade failure, preserve the upgrade logs, version packages, dependency check output, and current service status. Run continue or retry only after confirming that the continuation conditions are met.

Additional information

• Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
• Before a high-risk operation such as a database change, forced deletion, OSD or MON rebuild, pool deletion, service restart, or network change, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.