Issue/Introduction

This article describes how to troubleshoot 1 or more ZCenter services that repeatedly start, fail a health check, exit or are terminated, and start again after the VM hosting ZCenter is rebooted. Common causes include insufficient VM resources, disk I/O latency, dependent services that have not recovered, and restart policies enforced by an external health check or process manager.

Symptoms

• After the VM hosting ZCenter is rebooted, the management page remains inaccessible for a long time or intermittently returns 502/503.
• Java processes, containers, or Pods are repeatedly created and exit, and their PIDs keep changing.
• The systemd log contains Main process exited, Start request repeated too quickly, Failed with result, or start-limit-hit.
• A Docker environment reports Restarting, unhealthy, CrashLoopBackOff, Liveness probe failed, Readiness probe failed, or OOMKilled.
• The ZCenter log repeatedly shows service startup messages without reaching a stable running state. It might also report database, Redis, RabbitMQ, or Nacos connection timeouts, or a Flyway startup failure.
• The host has high I/O wait, insufficient memory, frequent swapping, sustained CPU saturation, or insufficient disk space.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: ZCenter Java services, Caddy, Gateway, Nacos, MariaDB/MySQL, Redis, and RabbitMQ
• Deployment: systemd, Docker, or Kubernetes on a VM

Possible causes

The issue can be caused by 1 or more of the following:

1. Insufficient disk performance: After a VM reboot, multiple services concurrently read JAR files, write logs, initialize database connections, or run Flyway. High disk latency can prevent services from becoming ready before the health-check deadline.
2. Insufficient CPU or CPU contention: In field cases, startup can take too long when fewer than 16 CPU cores are allocated. 16 cores is a field troubleshooting reference, not a confirmed official minimum for ZCenter 5.1.0 in the current materials.
3. Insufficient memory or swap thrashing: When multiple Java services start concurrently, insufficient available memory in the hosting VM can cause frequent GC, swapping, or operating system OOM termination. Memory must be measured in GB and cannot be described as “fewer than 16 cores”.
4. Slow recovery of dependent services: MariaDB/MySQL, Redis, RabbitMQ, or Nacos is not yet available, causing ZCenter connection timeouts, registration failures, or initialization failures.
5. Insufficient health-check grace period: External systemd, Docker, Kubernetes, or an operations platform considers the service unhealthy and restarts it before initialization finishes.
6. Incorrect startup order: ZCenter business services start at the same time as the database, Redis, RabbitMQ, and Nacos, so the business services initialize before their dependencies.
7. Disk capacity, inode, or file-system issue: A full disk, exhausted inodes, a read-only file system, or an I/O error prevents logs, upload directories, or the database from being written.
8. Database migration or configuration error: A deterministic error such as Flyway validation failure, database connection failure, or an incorrect Redis password causes every restart to fail at the same point. Adding resources alone cannot resolve this type of error.

Resolution

Check in the following order. Identify what restarts the service before determining whether resources or dependencies are the cause.

1. Identify the restarting object and restart source

• Check: Determine whether the entire VM is repeatedly rebooting or whether an external manager restarts a Java process, container, or Pod.

• Docker environment:

  docker ps -a
  docker inspect -f '{{.Name}} restart={{.HostConfig.RestartPolicy.Name}} count={{.RestartCount}} state={{json .State}}' <container>
  docker logs --since 2h <container>

• Expected result: Identify the service exit code, termination signal, restart policy, and health-check failure records.
• If abnormal: If no application exit stack exists and an external health check terminated the service, continue checking the health-check target, timeout, and startup grace period. If a consistent error stack exists, address that error first.

2. Check CPU, memory, and OOM events

• Commands:

  uptime
  free -m
  vmstat 1 10
  ps -eo pid,ppid,stat,etime,%cpu,%mem,rss,vsz,cmd --sort=-%cpu
  ps -eo pid,ppid,stat,etime,%cpu,%mem,rss,vsz,cmd --sort=-%mem
  dmesg -T --level=err,warn

• Expected result: CPU capacity remains available during startup, available memory is stable, swapping is not continuous, and there is no Out of memory, Killed process, OOMKilled, or exit code 137.
• If abnormal:
• If the CPU remains saturated and the only symptom is a startup timeout, increase CPU according to the business scale and official deployment specification, or stagger service startup. An allocation of fewer than 16 CPU cores is a condition worth investigating during field troubleshooting, not a substitute for formal capacity planning.
• If available memory is low, swapping is frequent, or OOM occurs, increase memory or adjust process resource limits based on actual Java process RSS, JVM heap configuration, and the official deployment specification.
• Do not merely increase the health-check timeout to conceal OOM or sustained CPU saturation.

3. Check disk capacity and I/O latency

• Commands:

  df -hT
  df -ih
  iostat -xz 1 10
  dmesg -T --level=err,warn

• Check: Focus on %util, await, %iowait, remaining disk capacity, and inodes.
• Expected result: Disk space and inodes are sufficient, I/O wait is not continuously high during startup, and there are no file-system or block-device errors.
• If abnormal:
• If disk capacity or inodes are insufficient, remove nonessential files or expand capacity according to operations procedures. Do not directly delete database or critical application directories.
• If await, %util, and %iowait remain high, check underlying storage performance, virtual disk type, host storage contention, and the snapshot chain.
• If the log contains No space left on device, Read-only file system, I/O error, or failure to create a log file, restore the file system and disk state first.

4. Check dependent services and startup order

• Check ports:

  ss -lntp | grep -E ':(3306|5672|6379|8848|8100|8200)\b'

• Check Nacos:

  curl -v --max-time 10 http://<nacos-host>:8848/nacos/actuator/health

• Check Redis:

  redis-cli -h <redis-host> -p <redis-port> --askpass PING

• Check: Confirm that MariaDB/MySQL, Redis, RabbitMQ, and Nacos have recovered before checking whether ZCenter services started and registered successfully.
• Expected result: Dependency ports are reachable, authentication succeeds, and Gateway can discover Portal, Base, and business service instances.
• If abnormal: If dependencies are not ready, restore them according to the deployment specification before starting ZCenter business services. Do not continuously restart every service in the hope that startup eventually succeeds.

5. Check service startup logs

• Log location: Usually logs/zscmp.log under the runtime directory. Use deployment parameters to determine the actual path.
• Recommended search terms:

  Application run failed
  OutOfMemoryError
  Killed process
  Flyway
  checksum mismatch
  HikariPool
  Unable to obtain connection
  Connection refused
  Redisson
  Nacos
  No space left on device
  I/O error
  Permission denied

• Decision:
• If every attempt exits at the same error stack, address that deterministic error first.
• If the failure point changes and high CPU, low memory, or high I/O wait also occurs, address the resource bottleneck first.
• If the service continues running but a health check terminates it, continue to step 6.

6. Check health checks and restart policies

• Check: Confirm that the health check accesses the target service's direct port, not an /actuator/** path that Gateway explicitly blocks.
• Note: Gateway returns 404 for /actuator/** requests that pass through it. If an external health check incorrectly accesses this path on Gateway, it might mark a healthy service as failed.
• Compare the health check's initial delay, timeout, interval, failure threshold, and startup probe with the actual startup duration.
• If abnormal: After ruling out insufficient resources and deterministic startup errors, appropriately increase the startup grace period or failure threshold. Do not permanently disable the health check.

7. Collect support information

• VM CPU, memory, swap, disk capacity, disk type, and I/O metrics at the time of failure.
• systemd/Docker/Kubernetes restart policy, health-check configuration, exit code, and restart count.
• A complete timeline from VM boot until services stabilize or exit again.
• ZCenter, database, Redis, RabbitMQ, Nacos, and Caddy logs from the same time window.
• Recent changes to VM specifications, disks, JVM arguments, service orchestration, or health-check configuration.

Handling

Choose the corresponding action based on the diagnosis. It is not recommended to change multiple variables at once:

1. Insufficient disk I/O: Migrate to storage that meets performance requirements, reduce host storage contention, handle an excessively long snapshot chain, and ensure that the disks containing log and database directories are healthy.
2. Insufficient CPU: Increase CPU according to business scale and official deployment specifications. Field observations indicate that startup might be too slow with fewer than 16 CPU cores; treat this value as a condition worth investigating during field troubleshooting, not as a capacity requirement.
3. Insufficient memory: Increase memory or correct container resource limits based on actual RSS, JVM heap, and OOM records. The current repository does not provide an official minimum-memory value.
4. Slow dependency recovery: Adjust startup order so that the database, Redis, RabbitMQ, and Nacos are ready before ZCenter services start.
5. Premature health check: After confirming that the service eventually starts successfully, adjust the initial delay, startup grace period, timeout, and failure threshold based on measured startup duration.
6. Deterministic configuration or migration error: Use the logs to correct database connectivity, Redis authentication, Nacos registration, or Flyway issues. Scaling resources alone cannot resolve these errors.

Verification

• Reboot: In a maintenance window, reboot the VM hosting ZCenter according to the deployment specification and confirm that every dependency and ZCenter service starts only once and remains stable.
• Process: Observe systemd, containers, or Pods for at least 1 complete health-check cycle and confirm that restart counts no longer increase.
• Resources: During startup, CPU, available memory, swap, and disk I/O remain within acceptable ranges, with no OOM or file-system errors.
• Ports: Ports 8100 and 8200 and dependency ports listen as required by the topology, and Gateway can discover business services.
• Functionality: Sign in, access the resource list, and run 1 controlled resource synchronization. The page no longer returns 502/503.
• Logs: Service logs no longer repeat the startup sequence or report health-check failures, dependency connection timeouts, or a fixed startup error.

Risks and rollback

• Adjusting CPU, memory, disk, or VM specifications usually requires a maintenance window. Confirm the business impact and VM rollback plan first.
• An overly permissive health check can delay detection of real failures. Preserve the original configuration and set values from measured startup duration instead of disabling health checks.
• Record the original systemd, Docker, or Kubernetes configuration before changing startup order or restart policy, and restore it if the change fails.
• Do not handle a restart loop by deleting a database, flushing Redis, deleting all logs, or repeatedly forcing service restarts.

Additional information

• For external access paths and ports, see How do I troubleshoot a ZCenter deployment or access failure caused by blocked ports?.
• For Redis connection failures, see How do I identify and recover Redis when ZCenter login or resource synchronization fails?.
• The current repository does not contain production systemd, Docker Compose, or ZCenter Kubernetes deployment files. Determine actual restart policies and health-check parameters from the target deployment configuration.