Issue/Introduction

This article describes how to troubleshoot database issues when Gateway remains accessible but Base, Portal, or a business service cannot start, an API fails, or intermittent timeouts occur. It focuses on database availability, authentication, connection pools, read-only state, and disk capacity. Use a separate KB to troubleshoot Flyway script validation issues.

Symptoms

• A service startup log contains Unable to obtain connection, Communications link failure, or Application run failed.
• An API returns an application failure, and the log contains HikariPool or Connection is not available, request timed out.
• The database returns Access denied for user, Unknown database, Too many connections, or a read-only error.
• The Gateway port is available, but the target service is not registered, returns 503, or fails to provide a business response.
• Database disk space or inodes are exhausted, or an InnoDB write fails.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: MariaDB/MySQL, HikariCP, Base, Portal, and business services

Possible causes

• The database process is not running, port 3306 is unreachable, or a firewall or route is abnormal.
• The address, database name, account, password, or host authorization in the effective application configuration is incorrect.
• The total connections from multiple ZCenter services and replicas exceed database max_connections.
• Slow queries, long transactions, lock waits, or slow database I/O exhaust the Hikari connection pool.
• An HA environment connects to a read-only replica, or the database has read_only/super_read_only enabled.
• The database data disk or temporary directory has insufficient free space, or the file system has insufficient available inodes.

Resolution

1. Confirm the scope and effective configuration

• Record the affected service, API, time, and error stack.
• Verify the database address, port, database name, and user in the service's effective Spring Profile, startup arguments, environment variables, and mounted configuration. Redact the password.
• Note: Gateway does not use the business database. A healthy Gateway does not mean that business service database connections are healthy.

2. Check the port and database process

ss -lntp | grep ':3306'
nc -vz <db-host> 3306
systemctl status mariadb
df -hT
df -ih

• Expected result: The database process runs, the port is reachable, and the data disk and inodes have sufficient free capacity.
• If abnormal: Restore the database service, network, or storage. Do not manually delete InnoDB files, redo logs, or the database directory.

3. Run read-only checks with the application account

mysql --protocol=TCP -h <db-host> -P <port> -u <application-user> -p

SELECT VERSION(), @@hostname, @@port;
SELECT USER(), CURRENT_USER(), DATABASE();
SELECT 1;
SELECT @@global.read_only, @@global.super_read_only;
SHOW VARIABLES WHERE Variable_name IN ('max_connections', 'read_only', 'super_read_only');
SHOW GLOBAL STATUS WHERE Variable_name IN ('Threads_connected', 'Threads_running', 'Max_used_connections', 'Aborted_connects');

• Expected result: Authentication succeeds over the same network path with the same account as the application. The target databases are visible, and a node that requires writes is the writable primary.
• Note: Availability of super_read_only depends on the database version. If the variable does not exist, use the read-only state variables supported by the current MariaDB/MySQL version.
• If abnormal: Correct account authorization, password, database name, or HA connection address according to the result. Do not blindly disable read-only mode on a replica.

4. Determine whether the connection pool is exhausted

• Compare Threads_connected, Max_used_connections, and max_connections.
• Check the number of ZCenter service instances and the Hikari pool limit for each instance. Replicas multiply the total connection requirement.
• Check database CPU, I/O, slow queries, long transactions, and lock waits, and correlate them with application thread dumps.
• Note: A Hikari timeout does not necessarily indicate a connection leak. A slow database or full disk can also hold connections.

5. Check application and database logs

• Search application logs for: HikariPool, Unable to acquire JDBC Connection, Too many connections, Access denied, read-only, and No space left on device.
• Search database logs for: InnoDB errors, refused connections, authentication failure, full disk, crash recovery, and primary/replica switchover.
• Also collect container standard output or systemd logs. Some ERROR entries might not be written to logs/zscmp.log.

Handling

1. If the database is unavailable, restore the database service and network first. After SELECT 1 succeeds, observe ZCenter service recovery.
2. If credentials are incorrect, modify the Secret or runtime configuration used by the deployment and verify account permissions over the same network path.
3. If connections are exhausted, first address slow queries, long transactions, connection storms, and database capacity. Do not blindly increase every Hikari pool and max_connections at the same time.
4. For a read-only error, restore the correct writable primary or VIP. Do not execute read_only=OFF on a replica whose role has not been confirmed.
5. If the disk is full, expand it or safely remove data according to database operations procedures. Do not manually delete core database files.
6. Hikari usually reconnects after the database recovers. If an individual service does not recover, restart only that service according to the deployment specification.

Verification

• Database-account authentication succeeds, and all required zscmp_* databases are visible.
• The writable node state is correct, and disk space and inodes are sufficient.
• Threads_connected remains stably below the limit, and no new Hikari connection timeout occurs.
• The affected services remain running and register with Nacos again.
• Complete 1 login, 1 read-only query, and 1 approved low-risk write operation.
• Check the business status in the response body and the service logs. Do not determine success solely from HTTP 200.

Risks and rollback

• Increasing a connection pool or database connection limit increases database memory pressure. Evaluate it against the number of instances.
• Changing an HA node role or read-only state can disrupt replication and must be performed by the database owner.
• Record the original connection parameters before changing them. If the change fails, restore the original configuration and restart only the affected services.

Additional information

• For port connectivity, see How do I troubleshoot a ZCenter deployment or access failure caused by blocked ports?.