Issue/Introduction

This article describes how to troubleshoot a host hang by using netconsole. It applies to the initial diagnosis and resolution of similar issues in ZStack Cloud.

Environment

• Product: ZCF
• Component: Cloud
• Version: 3.x-5.5.30

Symptoms

The host frequently hangs, and message or crash logs cannot be obtained. In this scenario, you can configure netconsole to send kernel logs to the management node for analysis of the hang.

Possible causes

After the host hangs, message or crash logs cannot be obtained.

Resolution

Configure netconsole to send kernel logs to the management node so that the cause of the hang can be analyzed.

1. Configure /etc/sysconfig/netconsole.

Add the following parameters to /etc/sysconfig/netconsole on the sending host:

  # Receiver IP address
  SYSLOGADDR=192.X.X.X
  # Port
  SYSLOGPORT=5888
  # Receiver MAC address
  SYSLOGMACADDR=b4:05:5d:aXXXX

2. Restart the netconsole service on the sending host.

  systemctl restart netconsole

3. Enable the netconsole service at startup.

  systemctl enable netconsole

4. Configure port 514 to listen on the receiving UDP port.

On the receiving host, uncomment $ModLoad imudp and $UDPServerRun 514 in /etc/rsyslog.conf, change 514 to the receiving UDP port, and run:

  sed -i 's/^#$Mod.*udp$/$ModLoad imudp/g;s/^#$UDP.*/$UDPServerRun 5888/g' /etc/rsyslog.conf

5. Restart the rsyslog service on the receiving host.

  systemctl restart rsyslog

6. Check whether the specified port is listening.

  lsof -i:5888

7. On the receiving host, check whether the system log contains system logs from the sending host. The common system log path is /var/log/messages. If the current environment uses a custom log path, use the actual path.

  tail -f /var/log/messages