Issue/Introduction

This article describes how to troubleshoot a cloud account or individual resource whose synchronization does not run, is skipped, or continuously reports a lock acquisition timeout even though Redis is generally available. This scenario indicates distributed-lock contention or a leftover lock and is not the same as a Redis connection failure.

Symptoms

• Only 1 cloud account or a small number of resources fail to synchronize, while other accounts work.
• A manual synchronization API returns success, but the target service does not enter the actual synchronization flow.
• The log contains resource sync get lock timeout!, one resource in cloud sync get lock timeout!, or Failed to acquire lock during resource sync.
• Redis PING succeeds, and login and other caching functions have no obvious issue.
• Synchronization was repeatedly submitted for the same account, or the issue began after the previous synchronization process exited unexpectedly.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: VM, RDS, OSS, Container, and Security resource synchronization, and Redis/Redisson

Possible causes

• The same account is being synchronized, and a new request exits after failing to acquire the account lock within 1 second.
• Another synchronization thread is processing the same resource, and the request still cannot acquire the resource lock after waiting about 120 seconds.
• The lock-holding process exited unexpectedly, but the lock remains within its fixed lease period.
• Synchronization runs longer than the lock lease, causing concurrent reentry or duplicate creation after the lock expires.
• Multiple instances use the same Redis database, and resources with the same cloudEnvId + originalId contend for the lock.
• lock timeout in the log means that lock acquisition returned false. It does not indicate a Redis network timeout.

Resolution

Lock key reference

• Account synchronization lock: account:sync:<serviceName>:<cloudAccountId>:.
• Individual resource lock: RSL:<cloudEnvId>:<originalId>.
• VM, OSS, Container, and Security account lock leases are usually up to about 600 seconds. An RDS account lock is up to about 1800 seconds.
• An individual resource lock waits about 120 seconds and has a fixed lease of about 30 seconds. Use the target version and runtime logs for actual behavior.

1. Stop repeated submissions and confirm the scope

• Record cloudAccountId, cloudEnvId, originalId, service name, synchronization start time, and all relevant instances.
• Stop repeatedly clicking synchronize to avoid more contention.
• Determine whether all resources for the account are blocked or only 1 resource times out.

2. Confirm Redis availability

redis-cli -h <redis-host> -p <redis-port> -n <db> --askpass PING

• Expected result: The command returns PONG.
• If abnormal: If Redis is unavailable, use the Redis connection failure KB. Do not continue treating the issue as an orphaned lock.

3. Find the exact lock key and TTL

redis-cli -h <redis-host> -p <redis-port> -n <db> --askpass --scan --pattern 'account:sync:*:<cloudAccountId>:'
redis-cli -h <redis-host> -p <redis-port> -n <db> --askpass --scan --pattern 'RSL:<cloudEnvId>:*'
redis-cli -h <redis-host> -p <redis-port> -n <db> --askpass TYPE '<exact-key>'
redis-cli -h <redis-host> -p <redis-port> -n <db> --askpass PTTL '<exact-key>'

• Expected result: The TTL of an active lock continuously decreases, and corresponding synchronization progress appears in the service log.
• Decision: PTTL=-2 means that the key does not exist. PTTL=-1 means that it has no expiration, which is abnormal for these fixed-lease locks.
• Note: Prefer SCAN. Do not run KEYS * in production Redis.

4. Correlate logs from all service instances

• Search for cloudAccountId, originalId, CloudAccountSyncHandler, resource sync get lock timeout, and synchronization completion messages.
• If the TTL decreases and the logs show ongoing progress, the lock still has an active holder. Wait for the task to finish.
• If the lock-holding process exited and no instance is continuing the work, wait for the fixed lease to expire naturally before testing again.
• Expiration of a 30-second resource lock does not prove that the original processing thread ended. Correlate the thread state and logs.

5. Rule out Redis connection failure and an untriggered task

• If multiple accounts fail simultaneously with Redisson connection errors, troubleshoot Redis.
• If there is no message-consumption log, check RabbitMQ and scheduled tasks. Do not misidentify an untriggered request as lock contention.

Handling

1. Stop repeated synchronization requests and wait for the running synchronization to finish.
2. If the lock-holding process exited, prefer waiting for the lease to expire naturally: observe a resource lock for at least 30 seconds, a regular account lock for up to about 600 seconds, and an RDS account lock for up to about 1800 seconds.
3. Handle 1 exact lock key only after proving that there is no active holder, the lease did not expire normally, and approval from operations has been obtained.
4. Deletion using wildcards, FLUSHDB, and bulk deletion of Redisson keys are prohibited.
5. If the lock is repeatedly recreated, check for duplicate scheduled tasks, duplicate triggers across instances, and synchronization runtime longer than the lease.

Verification

• Trigger 1 controlled single-account synchronization. Do not treat a successful API response as completion.
• Confirm that the target service receives the CloudAccountSyncHandler message and starts processing.
• Observe the account lock being created, its TTL decreasing, and its removal after completion.
• Confirm that no new lock acquisition timeout appears and that target resource data and synchronization state update.
• After 1 complete scheduled synchronization cycle, confirm that lock contention does not recur.

Risks and rollback

• Deleting an active lock allows concurrent synchronization and can cause resource-state overwrites, incorrect deletion, or inconsistent synchronization markers.
• A lock lease can end before business processing finishes. Do not infer that a task finished solely because the lock disappeared.
• If a lock is deleted accidentally, immediately stop new synchronization requests, preserve logs, and compare cloud-side resource state with ZCenter state.

Additional information

• For a general Redis failure, see How do I identify and recover Redis when ZCenter login or resource synchronization fails?.