VM HA does not trigger while a host is unresponsive but its storage heartbeat remains alive

Issue/Introduction

This article provides a structured troubleshooting process for a scenario where a host becomes unreachable, the VMs running on the host enter the unknown state in the UI, VM HA is not triggered, and services are interrupted.

Symptoms

• A host becomes unreachable through the management network.
• All VMs running on the host enter the unknown state in the UI and cannot be operated.
• Although the host is effectively unavailable to the platform, VM HA is not triggered.
• Hardware alarms are observed through out-of-band management, such as a disk backplane alarm.
• Restarting the HA services on the management nodes does not recover VM HA. It may also disrupt database synchronization between dual management nodes. The platform may need to temporarily fall back to single-management-node mode to restore UI access.

Environment

• Product: ZCF
• Versions: ["4.x", "5.x"]
• Components: ZStack Cloud
• Deployment form: Multi-host cluster with shared primary storage and dual management nodes.

Possible causes

• The host is in a false-death state: the OS is unresponsive but some kernel-level channels still respond. The storage heartbeat is still alive, so the HA preconditions are not met.
• Storage heartbeat detection did not break, causing the host checker to conclude that the host is still functional and preventing HA from triggering.
• The platform only triggers HA after both management-plane ping failure and storage-heartbeat failure; in a false-death state, only the management-plane check fails.
• The kvmagent process on the host is still partially responsive and reports the host as alive, even though it cannot serve the VMs on top.

Resolution

1. Confirm the host is unreachable through the management plane

• Check the management-node logs. Confirm the time when the platform first observed the host failure. Example observation from the source case: physical host 11 failed at 14:25:34; host self-check ping failed and the host was declared lost.
• Command (from a management node):

  zstack-cli LogInByAccount accountName=admin password='<password>'
  QueryHost uuid=b0d4f7a62d6c4f3e8a9c1b2d3e4f5a6b

• Confirm the host status changes to Disconnected in the UI.

2. Observe VM state change

• Confirm that the VMs on the failed host transition to unknown state in the UI.
• Command (list VMs on the affected host):

  zstack-cli LogInByAccount accountName=admin password='<password>'
  QueryVmInstance hostUuid=b0d4f7a62d6c4f3e8a9c1b2d3e4f5a6b

• Record the timestamp and the affected VM UUIDs.

3. Trace the HA Worker decision

• Inspect the management-node HA logs to confirm that the HA Worker was triggered for the failed host:

  grep "14:25" /usr/local/zstack/apache-tomcat/logs/management-server.log
  grep -i "b0d4f7a62d6c4f3e8a9c1b2d3e4f5a6b" /usr/local/zstack/apache-tomcat/logs/management-server.log

• The HA Worker invokes the host checker to evaluate HA preconditions. Inspect the host-checker log to determine whether the storage heartbeat check succeeded.
• Look for log signatures such as host checker success rate 0.83 exceeds threshold 0.5 to confirm the false-death decision.
• The HA decision is controlled by the host-checker success-rate threshold (> 0.5 in the source case). If the success rate is above the threshold, HA is not triggered.

4. Confirm the false-death state from the storage heartbeat and kvmagent

• Inspect the kvmagent log on the failed host (if accessible via out-of-band management) to determine whether the agent was actually killed, or whether it was stuck and unable to migrate or restart the VMs:

  tail -1000 /var/log/zstack/zstack-kvmagent.log | grep -iE "<vmUuid>|libvirt|qemu|kvm|destroy|kill|shutdown|poweroff|traceback|exception|error"

• The HA decision is based on the host-checker aggregate result; a single partial response from the storage heartbeat is enough to keep the success rate above the threshold and block HA.

5. Branch by host state

• If the host is in true death (storage heartbeat also fails): HA is expected to trigger automatically. If it does not, escalate to engineering as a possible HA Worker bug.
• If the host is in false death (storage heartbeat still responds but VMs are unreachable): HA will not trigger automatically. Proceed with the manual recovery actions below.

Manual recovery for the false-death branch

The source case uses the following manual recovery path.

1. Restart the HA management service to attempt to recover the affected VM state in the UI:

   zsha2 stop-node
   zsha2 start-node

2. If dual-MN database sync breaks during the recovery attempt (for example, because one management node has gone offline), the VIP UI service may fail to start. Switch to single-management-node mode by restarting services on the remaining management node and access the UI via the single-MN IP.
3. Once the UI is reachable in single-MN mode, force-stop the affected VMs' automatic HA recovery to stop the recovery loop.
4. After the failed host has been physically repaired and rebooted, verify the host reconnects to the platform and the VMs return to a normal state before re-enabling dual-MN mode.

Verification

• The affected host shows as Connected in the UI once it is physically restored.
• The VMs return to a normal state without stuck HA tasks.
• Dual-MN database sync resumes and the VIP UI service starts successfully.
• VM HA preconditions are re-evaluated correctly when the cluster is healthy again.

Additional information

• Recommended follow-up: Once the failed host is recovered, gather the kvmagent log from the time of failure to confirm whether the VMs were actually killed or were simply unable to be reached because the host was in a false-death state. This evidence can be used to confirm the root cause and improve future false-death handling.