Issue/Introduction

This article describes how to troubleshoot an order, resource operation record, resource synchronization, or page notification that is not updated after an asynchronous ZCenter task or cloud platform operation has finished. ZCenter uses RabbitMQ to transmit task-completion, order-state, resource-synchronization, and WebSocket notification messages. A RabbitMQ failure can prevent downstream business-state updates from being processed.

Symptoms

• A backend task is COMPLETED or FAILED, but the order remains in progress.
• A cloud platform operation has completed, but the ZCenter resource state, operation record, or page notification is not updated.
• A resource synchronization request has been submitted, but the corresponding business service does not start synchronization.
• RabbitMQ queue messages_ready continuously increases, or consumers=0 for a critical queue.
• The log contains AmqpConnectException, ACCESS_REFUSED, Failed to declare, Connection refused, [***error***], or TASK_END fail.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: RabbitMQ, Base, Portal, Operation, and resource services

Possible causes

• RabbitMQ is unavailable, the network is interrupted, or port 5672 is unreachable.
• The address, account, password, virtual host, or TLS parameters configured for ZCenter services are inconsistent.
• Queues and bindings exist, but the listener in the corresponding service did not start, leaving the consumer count at 0.
• Consumer processing fails, causing messages to be requeued or remain unacknowledged.
• A message was not published successfully during a RabbitMQ outage. Such a message is not necessarily republished automatically after recovery.
• Part of the business state is already committed to the database, so resubmission might cause duplicate creation, deletion, or billing.

Resolution

1. Confirm the business-state discrepancy

• Query the task, order, and actual cloud platform state. Record the task UUID, order UUID, resource UUID, and occurrence time.
• Task API example: GET /api/portal/v4/task/get?uuid=<TASK_UUID>.
• Order API example: GET /api/portal/v4/apply-order/getDetail?uuid=<ORDER_UUID>.
• Decision: If the task ended but the order remains in progress, focus on the TASK_END event. If resource synchronization never starts, focus on queues related to msg.resource.sync.

2. Check the RabbitMQ service and application connections

• Run on the RabbitMQ node:

  rabbitmq-diagnostics -q ping
  rabbitmq-diagnostics status
  rabbitmq-diagnostics listeners
  rabbitmqctl list_connections user vhost state channels peer_host peer_port

• Expected result: The RabbitMQ node is healthy, and connections from relevant ZCenter services exist and use the correct vhost.
• If abnormal: Restore RabbitMQ and network connectivity, and verify the runtime address, account, vhost, and TLS settings. Do not use example credentials from the source code to assess a production configuration.

3. Check queues, consumers, and backlog

• Commands:

  rabbitmqctl list_queues -p <vhost> name messages_ready messages_unacknowledged consumers
  rabbitmqctl list_bindings -p <vhost> source_name destination_name destination_kind routing_key

• Key queues: inner_base, inner_operation, inner_vm, base_log, mo_sync_vm, order_vm, order_oc_vm, and ws_<portal-instance>.
• Expected result: Critical queues exist, relevant consumer counts are greater than 0, and the message backlog continuously decreases.
• If abnormal: If a consumer count is 0, check the corresponding service and listener. If messages_ready grows, check consumer logs. If messages_unacknowledged does not decrease, locate the consumer-processing error.

4. Check ZCenter logs

• Search for: [base TASK_END], [oc TASK_END], [***pre***], [***after***], [***error***], org.springframework.amqp, CachingConnectionFactory, and ACCESS_REFUSED.
• Decision: If a pre-publish log exists but no consumption log follows, check the publishing connection and routing. If a pre-consumption log exists but no post-consumption log follows, check the handler error. If no message log exists, first confirm whether the upstream business actually sent the message.

5. Collect support information

• RabbitMQ node, vhost, queue, consumer, backlog, and connection information, with account passwords redacted.
• Base, Operation, Portal, and target resource service logs from the same time window.
• Comparison of task, order, and cloud platform states, clearly identifying completed steps.

Handling

1. Restore the RabbitMQ node, network, account permissions, and vhost, and confirm that application connections are reestablished.
2. If a queue exists but has no consumer, restore only the corresponding service. Do not indiscriminately restart every component.
3. After RabbitMQ recovers, observe the backlog being consumed naturally. Do not purge a queue or delete the vhost.
4. For orders and resource operations stuck during the outage, compare each item with the actual cloud-side result before following the product recovery process and business idempotency requirements.
5. Do not bulk-resubmit tasks. A message that was not published successfully during a connection outage cannot be assumed to be republished automatically.

Verification

• RabbitMQ: The node is healthy, application connections exist, and critical queue consumer counts are greater than 0.
• Backlog: messages_ready and messages_unacknowledged return to their normal ranges.
• Task: Create 1 approved low-risk task and confirm that it reaches a final state.
• Order: The order task record matches the final state and no longer remains in progress.
• Logs: Paired pre- and post-consumption logs appear, and no new AMQP connection or authentication error is reported.

Risks and rollback

• Purging queues, deleting a vhost, or rebuilding the RabbitMQ data directory can cause permanent message loss and is prohibited as a routine recovery method.
• The cloud-side operation might already have succeeded, so resubmission can create resource or billing risk.
• Record the original RabbitMQ connection parameters before changing them. If the change fails, restore the original configuration and restart only the affected services.

Additional information

• For RabbitMQ ports and dependency connectivity, see How do I troubleshoot a ZCenter deployment or access failure caused by blocked ports?.