Issue/Introduction

This article describes how to troubleshoot login authentication, permission retrieval, and resource synchronization issues caused by an unreachable Redis instance, authentication failure, connection timeout, or Redis service failure. Different services might return different results for Redis errors. Use connection tests and service logs to determine the cause.

Symptoms

• Login, token refresh, or access to an authenticated API fails, possibly with HTTP 401. The log might contain Token expired, Token validation failed, Token verification failed, or Auth error.
• Cloud environment resource synchronization does not run, a synchronization task keeps retrying, or synchronization of a single resource times out. The log might contain resource sync get lock timeout! or Failed to acquire lock during resource sync.
• Role permissions cannot be retrieved correctly, the page reports insufficient permission, or login-lockout behavior is abnormal.
• Service startup or runtime logs contain client errors such as Redis connection refused, timeout, or authentication failure. Use the actual client logs for the exact error text.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: Base, Gateway, and resource synchronization services that use Redis/Redisson
• Deployment: Single-node or multi-service deployment. Use the Redis address and password from the actual runtime configuration.

Possible causes

• The Redis process is not running, or its listen address or port is unreachable.
• The Redis address, port, password, database, or TLS mode in the service configuration does not match the actual deployment.
• Redis connection count, resources, or network conditions are abnormal, causing Redisson connection timeouts or retry failures.
• A Redis failure prevents access token keys, refresh token keys, or distributed locks from being read or written. Resource synchronization might be unable to acquire a lock.
• If only permissions are abnormal, the issue might instead involve permission configuration, cache refresh, or the calling service. A page symptom alone is not sufficient to identify a Redis failure.

Resolution

1. Confirm the scope and time

• Record the first occurrence, affected users, cloud environments, resources, and APIs.
• Determine whether all requests fail, 1 service fails, or only resource synchronization or permission display is abnormal.
• If only 1 user or resource is affected, preserve the corresponding user ID, cloudAccountId, resourceId, and request time before attempting a global recovery operation.

2. Check connectivity between the application and Redis

• Check: Access the actual Redis address from the network namespace of the affected service.
• Command:

  redis-cli -h <redis-host> -p <redis-port> --askpass PING

• Expected result: The command returns PONG, and the address, port, password, and database match the runtime configuration.
• If abnormal: Check the Redis process, listening port, firewall/security group, network route, and credentials separately. Do not use an example address or password from the source code.

3. Check service health and logs

• Command/UI path: Access /actuator/health on the affected service, and check the logs of Base, Gateway, and the corresponding business service.
• Recommended search terms: Token validation failed, Token expired, Token verification failed, Auth error, resource sync get lock timeout, Failed to acquire lock during resource sync, and tryLock fail.
• Expected result: The service health check is normal, and the logs no longer continuously report Redis connection errors or lock acquisition timeouts.
• If abnormal: Correlate the failure time with Redis service logs, connection count, and resource usage. Do not assume that every Redis operation is healthy solely from the health endpoint.

4. Check the affected functional path

• If login and authentication both fail: Check whether the base:access_token:{userId}:{jti} and base:refresh_token:{userId}:{jti} keys created by a new login can be written and read, and use the new token to access an authenticated API.
• If only resource synchronization fails: Confirm that the cloud account synchronization request was initiated, and then check whether the corresponding service continuously reports distributed-lock timeouts. Do not manually delete a lock key while synchronization is still running.
• If only permissions are abnormal: Check role permission configuration and cache refresh, and verify Base and Redis reads and writes. The source code shows scheduled refresh and database fallback for the role-permission cache, so do not conclude that every permission request must fail.
• If only login lockout is abnormal: Check the base:loginFailCount:{userId} and base:loginLock:{userId} keys and their TTLs for a test user. Do not unlock the user by flushing the database.

5. Collect support information

• Redis status, deployment topology, actual connection address, port, database, and whether high availability is enabled.
• Affected service and Redis logs covering at least 1 complete time window before and after the failure.
• Affected user, cloudAccountId, resourceId, request URL, HTTP status code, and exact timestamp.
• Change records, including Redis restarts, failovers, password changes, network policy changes, and ZCenter service restarts.

Handling

1. If Redis is stopped, restore it or perform an approved failover according to the deployment specification. Confirm port and authentication availability before resuming ZCenter operations.
2. If the configuration is inconsistent, correct the Redis address, port, password, database, and TLS configuration in the actual runtime configuration, and restart the affected service as required by the deployment. Do not modify source files instead of runtime configuration.
3. If a network or firewall rule blocks access, allow only the required path from the affected service to Redis and preserve connectivity results from before and after the change.
4. After Redis recovers, verify login and an authenticated API first, and then run 1 controlled cloud account resource synchronization. Do not delete all tokens or distributed-lock keys.

Verification

• Redis: Run a read-only connection test from the network location of every service that actually uses Redis, and confirm that it returns PONG.
• Authentication: Sign in again, access an authenticated API, refresh the token, and verify that the token cannot be used after logout.
• Permissions: Use a test user to access an API with a known role permission, and confirm that the result matches the configuration.
• Synchronization: Trigger synchronization for 1 controlled cloud account, confirm that the task completes and resource state updates, and verify that lock timeouts no longer appear in the log.
• Alarms and logs: Monitor for 1 complete business cycle and confirm that connection timeouts, authentication failures, and lock acquisition failures do not recur.

Risks and rollback

• A Redis restart or failover can briefly interrupt authentication, caching, and synchronization. Perform it in a maintenance window and notify users in advance.
• Do not use FLUSHDB, delete all token keys by prefix, or bulk-delete Redisson locks as a routine recovery action. Doing so can invalidate all sessions or disrupt running synchronization tasks.
• If a configuration change fails, restore the previous runtime configuration and restart only the affected services as required by the deployment. Preserve Redis and application logs for investigation.

Additional information

• Token keys: base:access_token:{userId}:{jti} and base:refresh_token:{userId}:{jti}.
• Login-lockout keys: base:loginFailCount:{userId} and base:loginLock:{userId}.
• The exact key format and wait time of a resource synchronization lock are determined by the business service. Use cloudAccountId/resourceId from the log to locate the issue.
• Related KB: How do I safely adjust the ZCenter token expiration time when it does not meet business requirements?.