How do I troubleshoot repeated ZCenter service restarts after rebooting its VM?

Issue/Introduction

This article describes how to troubleshoot 1 or more ZCenter services that repeatedly start, fail a health check, exit or are terminated, and start again after the VM hosting ZCenter is rebooted. Common causes include insufficient VM resources, disk I/O latency, dependent services that have not recovered, and restart policies enforced by an external health check or process manager.

Symptoms

• After the VM hosting ZCenter is rebooted, the management page remains inaccessible for a long time or intermittently returns 502/503.
• Java processes, containers, or Pods are repeatedly created and exit, and their PIDs keep changing.
• The systemd log contains Main process exited, Start request repeated too quickly, Failed with result, or start-limit-hit.
• A Docker environment reports Restarting, unhealthy, CrashLoopBackOff, Liveness probe failed, Readiness probe failed, or OOMKilled.
• The ZCenter log repeatedly shows service startup messages without reaching a stable running state. It might also report database, Redis, RabbitMQ, or Nacos connection timeouts, or a Flyway startup failure.
• The host has high I/O wait, insufficient memory, frequent swapping, sustained CPU saturation, or insufficient disk space.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: ZCenter Java services, Caddy, Gateway, Nacos, MariaDB/MySQL, Redis, and RabbitMQ
• Deployment: systemd, Docker, or Kubernetes on a VM

Possible causes

The issue can be caused by 1 or more of the following:

1. Insufficient disk performance