DocsKnowledge BaseSecurityAPI Explorer
Documentation›KB›Troubleshoot ZStack Cloud elastic BareMetal instance issuesKB
Document navigation

Issue/Introduction

This article applies to failures when adding a gateway node or BareMetal node, obtaining hardware information, booting through PXE, creating an instance, opening a console, attaching a volume, or obtaining the Ready status for elastic BareMetal. Check the management node, gateway node, provision network, IPMI, BIOS, PXE or TFTP, bare metal agent, iSCSI, and components in the guest operating system together.

Symptoms

  • Adding a gateway node fails and reports that a LUN is not loaded or storage access is abnormal.
  • Adding a BareMetal node fails and reports an incorrect IPMI user name or password, a timeout, or an inability to establish an IPMI session.
  • Obtaining hardware information fails, PXE boot becomes stuck, or Failed to load ldlinux.c32 is reported.
  • PXE boot fails, and bare metal cannot obtain a provision network address from the gateway node or access TFTP.
  • Opening the console of a Windows elastic BareMetal instance fails and reports open remote baremetal instance console port failed.
  • Adding a volume to elastic BareMetal fails and reports that no iSCSI session exists.
  • Obtaining the Ready status fails and reports 502 Bad Gateway.

Environment

  • Product: ZCF
  • Component: Cloud
  • Affected objects: ZStack Cloud; elastic BareMetal, gateway node, provision network, IPMI, PXE, TFTP, iSCSI, VNC Server, and bare metal agent
  • Version: 4.x-4.8.38

Possible causes

  • A SAN primary storage LUN is mapped only to compute nodes and not to the elastic BareMetal gateway node, causing gateway node addition or storage access to fail.
  • The gateway node provision network and bare metal PXE NIC are not on the same L2 network or VLAN, or the BareMetal node is also connected to another DHCP network, causing PXE to obtain the wrong address.
  • A residual IP address exists on a gateway node NIC, bond, or bridge, and the NIC bound to the provision network on the platform differs from the address providing DHCP or TFTP, preventing the BareMetal node from obtaining a provision network address or accessing TFTP.
  • The network from the management node to bare metal IPMI is unreachable or has severe packet loss, bidirectional UDP port 623 access is not allowed, or IPMI over LAN is not enabled in the BIOS, causing BareMetal node addition to report an incorrect IP address, port, user name, or password, or a session timeout.
  • The IPMI password or BMC information was changed at the underlying layer but not synchronized to the platform, causing an unknown BareMetal node power status or the instance Ready status to be Disconnected.
  • PXE or UEFI boot is not enabled for the target NIC in the bare metal BIOS, or the server has multiple in-band and out-of-band IPMI addresses and IPMI network sharelink is not disabled, causing the IPMI address returned during hardware information collection to differ from the platform record.
  • iPXE, vmlinuz, initrd.img, NIC firmware, or the NIC driver is incompatible with the server hardware, resulting in Failed to load ldlinux.c32, x86_64/vmlinuz no such file or directory, no more network device, a hang after initrd loads, or missing NIC firmware such as bnx2x.
  • The bare metal instance image was not created according to elastic BareMetal requirements and lacks the bare metal agent, Python runtime, open-iscsi or iSCSI components, or the agent version has a known compatibility issue, causing Ready status, volume attachment, or probe failures.
  • The provision network IP address inside the bare metal instance differs from the address assigned by the platform, or an imported existing operating system did not reacquire a provision network address through DHCP, preventing the gateway from reaching the instance agent over the provision network.
  • The Windows elastic BareMetal image lacks VNC Server or the bare metal agent, causing console access to fail.
  • Backend storage is being changed, the iSCSI target is unavailable, or the instance and gateway node cannot establish an iSCSI session, causing a Ready status timeout, volume attachment failure, or I/O failure after instance startup.

Resolution

1. Identify the failure stage

  • Record the bare metal instance UUID, gateway node UUID, BareMetal node IPMI address, provision network, storage type, and failure time.
  • Determine whether the issue occurs while adding a gateway node, adding a BareMetal node, obtaining hardware information, creating an instance, accessing the console, attaching a volume, or probing readiness.
  • For a PXE or hardware information issue, it is recommended that you open the IPMI console and observe the real-time output after the BareMetal node boots from its NIC.

2. Check the gateway node, storage mappings, and provision network

If adding a gateway node fails or PXE cannot obtain a provision network address, first confirm the gateway node prerequisites.

ip addr
ip route
bridge link
systemctl status zstack-baremetal-tftpd
ss -lunpt | egrep "dhcp|tftp|:69"
  • If SAN primary storage is used, confirm that the primary storage LUN is also mapped to the gateway node. Mapping it only to compute nodes causes gateway node addition to fail or the LUN not to load.
  • Confirm that the gateway node IP address entered on the platform matches the actual provision network IP address of the gateway node.
  • Confirm that the gateway node provision network NIC and the bare metal PXE NIC are on the same L2 network or VLAN.
  • Confirm that no residual IP address exists on the gateway node bond or physical NIC. If the platform creates a bridge based on the bond but an IP address remains on the bond or an old bridge, DHCP or TFTP might bind to the wrong address.
  • If the provision network is configured incorrectly, follow the environment change process to remove residual addresses and then reconfigure the provision network IP address on the platform. Afterward, add or reconnect the gateway node again.

3. Check IPMI, BMC, and BIOS settings

ping <ipmi-ip>
ipmitool -I lanplus -H <ipmi-ip> -U <user> -P '<password>' chassis power status
ipmitool -I lanplus -H <ipmi-ip> -U <user> -P '<password>' lan print 1
  • Confirm that the network from the management node to the IPMI address is stable and has no significant packet loss.
  • Check whether IPMI over LAN and PXE or UEFI boot for the target NIC are enabled in the BIOS.
  • If ipmitool reports Unable to establish IPMI v1.5/RMCP session, Unable to establish LAN session, or a timeout, check the bidirectional UDP port 623 access policy, especially whether return traffic from UDP 623 -> 10240-65535 is allowed.
  • If the server has both in-band and out-of-band IPMI addresses, disable the unused shared network or IPMI network sharelink to prevent hardware information collection from recognizing an IPMI address that differs from the platform record.
  • If the IPMI password or BMC information was changed at the underlying layer, synchronize the BareMetal node IPMI information on the platform. If an older version does not support changing it in the UI, update it through a controlled CLI or API method.

4. Check DHCP and TFTP downloads in the first PXE stage

tail -n 200 /var/log/zstack/baremetalv2/dnsmasq.log
tail -n 200 /var/log/messages
ls -l /var/lib/zstack/baremetalv2/tftpboot/
ls -l /var/lib/zstack/baremetalv2/tftpboot/x86_64/
  • In dnsmasq.log, check whether the MAC address in the DHCP offer belongs to the bare metal PXE NIC.
  • If multiple MAC addresses appear, confirm whether bare metal is connected to another DHCP network or a non-PXE NIC is interfering with boot.
  • In /var/log/messages, check TFTP access records and confirm whether ipxe.efi, boot.ipxe, x86_64/vmlinuz, and x86_64/initrd.img are requested and downloaded normally.
  • If Failed to load ldlinux.c32 is reported, first confirm the current boot mode. It is generally recommended that you switch to UEFI boot and retest.
  • If x86_64/vmlinuz no such file or directory is reported, confirm whether the files under tftpboot/x86_64/ exist and have correct permissions, and whether the gateway node was reconnected or the bare metal service was redeployed.

5. Check the second PXE stage and hardware compatibility

  • Stage 1: The NIC firmware obtains an address from DHCP for PXE.
  • Stage 2: After vmlinuz and initrd.img load, the temporary Linux system obtains an address again and returns hardware information.
  • Success in stage 1 does not mean that stage 2 succeeds. If stage 2 cannot obtain an address, check the initrd driver, NIC compatibility, DHCP or TFTP service, and switch VLAN.
  • If startup hangs after initrd loads or the console reports missing NIC firmware or drivers such as bnx2x or ngbe, initrd generally lacks the corresponding hardware driver. Collect the server model, NIC model, firmware version, console screenshot, and gateway node logs, and then contact ZStack Technical Support to assess driver or boot file adaptation.
  • If the console reports no more network device, you can assess using a newer iPXE or upgrading the NIC firmware after confirmation from ZStack Technical Support.
  • If the server has a special graphics card, new NIC, Hygon CPU, or vendor firmware compatibility issue, open the IPMI console to record where startup hangs and first contact the hardware vendor to confirm a firmware upgrade plan.
  • It is not recommended that you replace ipxe.efi, vmlinuz, or initrd.img yourself before confirming the version and hardware model. If replacement is required, first back up the original file and have ZStack Technical Support confirm the source of the replacement package and the rollback method.

6. Check instance creation failures and the image template

  • Before creating an instance, confirm that the bare metal image was created according to elastic BareMetal requirements and includes the bare metal agent, required network configuration, iSCSI or open-iscsi components, and operating system dependencies.
  • If creating an instance reports failed to create provision configurations for baremetal2 instance, check whether the gateway node has the required configuration for the storage type. When specific storage such as enterprise-edition ZCE is used, confirm whether the gateway node has the corresponding xdc configuration.
  • If the instance cannot enter the operating system after startup, check initramfs, NIC naming, root volume recognition, iSCSI boot dependencies, and system logs in the image.

7. Check the Ready status, agent, and provision network connectivity

When obtaining the Ready status fails, first troubleshoot the path from the MN through gateway node Nginx to the bare metal instance agent.

# Inside the bare metal instance
systemctl status zstack-baremetal-instance-agent.service
journalctl -u zstack-baremetal-instance-agent.service -n 100 --no-pager
tail -n 200 /var/log/zstack/baremetal2/bm-instance-agent.log
ip addr
ss -tlnp
# On the gateway node
tail -n 100 /var/log/nginx/error.log
curl -v http://<instance-provision-ip>:<agent-port>/
  • If 502 Bad Gateway or BM.1001 nginx connection error is reported, the management node has generally reached gateway node Nginx, but the upstream connection from Nginx to the instance agent is unavailable. Use the Nginx error log to distinguish Connection refused, timed out, no live upstreams, and configuration errors.
  • If the instance agent is not running, check the Python runtime, agent installation status, and service logs. If a Python dependency is missing, install it and restart the agent.
  • If the agent is normal but access from the gateway times out, check whether the instance provision network IP address matches the address assigned by the platform. If necessary, reacquire a provision network address through DHCP inside the instance.
  • In an imported operating system, if the instance still uses the old provision network address, you can run dhclient inside the system to reacquire an IP address. After confirming that it works, assess whether to add it to a startup script.
  • If failed to ping baremetal2 instance through gateway is reported and the logs contain an iSCSI error, check whether open-iscsi or iSCSI components are installed and running in the instance.
  • If a Ready probe times out after the gateway node is reconnected, you can disable the affected elastic BareMetal resources, reconnect the gateway node, and enable the resources incrementally for verification when there is no production workload impact.

8. Check the console, agent, and VNC Server

  • The console of a Windows elastic BareMetal instance depends on the agent and VNC Server, such as TigerVNC. If open remote baremetal instance console port failed is reported, first confirm whether VNC Server is installed in the image template and configured to start automatically.
  • Check Windows Firewall, the VNC service listening port, and the instance agent status.
  • If the out-of-band console of a BareMetal node opens with no content, confirm whether the server BMC requires HTTPS access. Some models do not automatically redirect HTTP to HTTPS, so you must manually use HTTPS to access the out-of-band management UI.

9. Check volume attachment and iSCSI sessions

# Inside the bare metal instance
systemctl status iscsid
iscsiadm -m session
iscsiadm -m node
rpm -qa | egrep "iscsi|open-iscsi"
  • If adding a volume fails and reports no iSCSI session, first confirm whether iSCSI components are installed in the instance, iscsid is running, and the network from the instance to the gateway node or storage network is reachable.
  • If the instance can see the gateway node iSCSI session at the underlying layer but the platform still reports no session, check the bare metal agent version. An older agent might have a session recognition issue and must be upgraded to a version that matches the current cloud platform.
  • If another target or unknown disk appears in the instance, check whether open-iscsi automatically signs in to a historical target at startup. If necessary, clean unrelated session or node records and adjust the node.startup policy.
  • If storage is being changed or the iSCSI target is unstable, wait until the storage change completes before probing readiness, attaching a volume, or starting the instance.

10. Verify recovery

zstack-ctl status
systemctl status zstack-baremetal-tftpd
  • Obtain the BareMetal node hardware information again and confirm in the IPMI console that PXE, TFTP, vmlinuz, and initrd load normally.
  • Create or retry the elastic BareMetal instance and confirm that it can enter the operating system.
  • Inside the instance, confirm that the agent, VNC Server, iscsid, and network address status are normal.
  • On the platform, obtain the Ready status again, open the console, attach a volume, and record the recovery time and verification result.

Precautions

  • Base the troubleshooting conclusion on the current version, service status, logs, cluster health, recent changes, and on-site impact.
  • Before a high-risk operation such as reconnecting a gateway node, changing the provision network, replacing iPXE, vmlinuz, or initrd, upgrading the agent, cleaning an iSCSI session, changing storage configuration, or modifying the database, complete a backup, obtain approval, and obtain confirmation from ZStack Technical Support.
  • This information provides troubleshooting guidance only and does not replace current environment logs, console output, network packet captures, and version compatibility confirmation.
PreviousWhat can I do if pressing Ctrl+Alt+Del causes a host to disconnect?NextTroubleshoot ZStack Cloud GPU and vGPU resource issues
Troubleshoot ZStack Cloud elastic BareMetal instance issues | 故障排查 · KB | ZStack Resource Center