How do I troubleshoot RabbitMQ when an order or operation remains in progress after its task finishes?

Issue/Introduction

This article describes how to troubleshoot an order, resource operation record, resource synchronization, or page notification that is not updated after an asynchronous ZCenter task or cloud platform operation has finished. ZCenter uses RabbitMQ to transmit task-completion, order-state, resource-synchronization, and WebSocket notification messages. A RabbitMQ failure can prevent downstream business-state updates from being processed.

Symptoms

• A backend task is COMPLETED or FAILED, but the order remains in progress.
• A cloud platform operation has completed, but the ZCenter resource state, operation record, or page notification is not updated.
• A resource synchronization request has been submitted, but the corresponding business service does not start synchronization.
• RabbitMQ queue messages_ready continuously increases, or consumers=0 for a critical queue.
• The log contains AmqpConnectException, ACCESS_REFUSED, Failed to declare, Connection refused, [***error***], or TASK_END fail.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: RabbitMQ, Base, Portal, Operation, and resource services

Possible causes

• RabbitMQ is unavailable, the network is interrupted, or port 5672 is unreachable.
• The address, account, password, virtual host, or TLS parameters configured for ZCenter services are inconsistent.