Issue/Introduction

This article describes how to troubleshoot an inaccessible ZCenter page, an API that returns 502/503, a browser certificate error, or a certificate that is not updated after HTTPS is enabled or an SSL certificate is uploaded. After ZCenter saves sslSetting, it asynchronously notifies Base through RabbitMQ to update Caddy. A successful save does not mean that Caddy loaded the configuration successfully.

Symptoms

• After HTTPS is enabled, the browser cannot connect, the TLS handshake fails, or an incorrect certificate is returned.
• The certificate upload API succeeds, but access still presents the old or built-in certificate.
• The HTTP endpoint works but the HTTPS endpoint does not, or the setting shows HTTPS as enabled while Caddy still listens in HTTP mode.
• Base or Caddy logs contain errors related to certificate loading, the Caddy admin API, MQ message consumption, or port listening.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: Portal global settings, Base, RabbitMQ, Caddy, and SSL certificates

Possible causes

• sslSetting has been written to the database, but the RabbitMQ refresh event was not sent or consumed successfully.
• Base cannot access the Caddy admin API at 127.0.0.1:2019.
• The certificate and private key do not match, the PEM format is incorrect, the private key is encrypted, the certificate chain is incomplete, or file permissions are insufficient.
• Caddy dynamic ports 8300/8301 do not match the external load balancer or port mapping.
• The certificate SAN does not match the access domain, or the built-in certificate uses an unexpected host IP address.
• Only part of the Caddy configuration was updated, leaving the database state inconsistent with the running configuration.

Resolution

1. Preserve an HTTP recovery path and confirm the base endpoint

curl -v --max-time 10 http://127.0.0.1:8100/
curl -v --max-time 10 http://127.0.0.1:8100/api/portal/v4/pub/login

• Expected result: The base endpoint on port 8100 remains accessible.
• If abnormal: If port 8100 is also unreachable, troubleshoot the base Caddy/Gateway services and ports before changing the certificate further.

2. Check listening ports and the Caddy admin API

ss -lntp | grep -E ':(8100|8200|8300|8301|2019)\b'
curl -sS http://127.0.0.1:2019/config/apps/http/servers/srv3
curl -sS http://127.0.0.1:2019/config/apps/tls

• Expected result: When HTTPS is enabled, the dynamic service listens on port 8301. In HTTP mode, it listens on port 8300. The Caddy admin API is accessible only locally.
• If abnormal: If port 2019 is unreachable, check the Caddy process and admin API. If the dynamic state differs from the database, check the Base refresh event.

3. Check the global setting and refresh event

• Query the current value of GLOBAL_SETTINGS/sslSetting, and search the Base log for REFRESH_SSL_CONFIG, [***error***], and Caddy update failures.
• Also check the RabbitMQ inner_base queue and its consumers.
• Decision: If the save API succeeds but there is no refresh-event consumption record, troubleshoot RabbitMQ or the Base listener first.

4. Validate the certificate, private key, and certificate chain

openssl x509 -in /var/lib/caddy/tls/tls.crt -noout -subject -issuer -serial -dates -fingerprint -sha256
openssl pkey -in /var/lib/caddy/tls/tls.key -noout -check
openssl x509 -in /var/lib/caddy/tls/tls.crt -pubkey -noout | openssl pkey -pubin -outform DER | openssl sha256
openssl pkey -in /var/lib/caddy/tls/tls.key -pubout -outform DER | openssl sha256

• Expected result: The certificate is valid, the private key is readable, the 2 public-key digests match, and the certificate chain and SAN meet the access-domain requirements.
• If abnormal: Prepare a correct PEM certificate and unencrypted private key, and correct the file permissions. Do not assume that the certificate is valid only because the upload API succeeded.

5. Check the certificate presented externally

openssl s_client -connect <host>:<public-port> -servername <certificate-dns-name> -showcerts </dev/null

• Expected result: The target certificate and complete chain are returned, and the domain matches the SAN.
• If abnormal: Check the load balancer, VIP, SNI, and port mapping, and confirm that the request reaches the intended Caddy instance.

Handling

1. For a certificate issue, correct the certificate, private key, chain, and file permissions, upload the certificate again, and trigger an SSL configuration refresh.
2. If RabbitMQ or Base did not consume the refresh event, restore the message path and save sslSetting again to trigger an update.
3. If the Caddy admin API is unavailable, restore Caddy according to the deployment specification. Do not modify only the database setting.
4. For quick recovery, use the still-accessible HTTP endpoint to restore sslSetting to {"enable":false}, and verify that Caddy switches back to the HTTP template.
5. Correct the actual mapping from the external load balancer or VIP to port 8300 or 8301. Use the public port from the target deployment.

Verification

• Configuration: The sslSetting value in the database matches the expected setting.
• Caddy: The dynamic service and TLS objects in the admin API are updated, and the listening port matches the mode.
• Certificate: openssl s_client returns the expected certificate, chain, and SNI result.
• Business: The page, static resources, and at least 1 API request are accessible through the target HTTPS address.
• Logs: Base and Caddy no longer report certificate-loading, admin-API, or TLS-handshake errors.

Risks and rollback

• The SSL switch is a global setting, so a change might affect all users and API clients.
• Directly modifying the database does not trigger a Caddy refresh and might leave the configuration inconsistent for an extended period.
• Deleting certificate files might leave an inconsistent state in which only the certificate or private-key file remains and should not be the preferred rollback method.
• Before making changes, preserve the original sslSetting, certificate fingerprint, and Caddy state. If the change fails, restore the original setting and certificate through the HTTP path.

Additional information

• For ports and the proxy path, see How do I troubleshoot a ZCenter deployment or access failure caused by blocked ports?.
• For RabbitMQ refresh-event troubleshooting, see How do I troubleshoot RabbitMQ when an order or operation remains in progress after its task finishes?.