Issue/Introduction

This article describes how to troubleshoot HTTP 503 errors, no available service instance messages, or LoadBalancer service discovery failures when Gateway itself is accessible but a Portal, VM, or other business API is not. This scenario focuses on Nacos registration and discovery and does not repeat connectivity troubleshooting for external port 8100.

Symptoms

• The Gateway port accepts connections, but /portal/..., /vm/..., or another business route returns HTTP 503.
• The error contains No servers available, no available service instance, LoadBalancer, or a service discovery timeout.
• The target service process appears to run, but it is absent, unhealthy, or disabled in Nacos, or its registered IP address and port are unreachable.
• Gateway and the target service use different Nacos addresses, namespaces, or service names, making the target service invisible to Gateway.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: Spring Cloud Gateway, Spring Cloud LoadBalancer, Nacos, and target business services
• Deployment: Single-node, multi-node, or container deployment

Possible causes

• Nacos is unreachable, authentication fails, or the target service is not registered.
• The target business service is not running or is in the starting or exited state, so Nacos has no available instance.
• Gateway and the target service use different Nacos addresses, namespaces, or groups.
• The registered service name differs from the name expected by the Gateway route. For example, the Gateway /portal/** route maps to app-portal.
• The registered instance IP address or port is incorrect, or a container registered an address unreachable from another network namespace.
• An instance exists but has healthy=false, enabled=false, or has expired.
• The request path is incorrect. Gateway removes the first path segment before forwarding. For example, /portal/v4/pub/login is forwarded to the target as /v4/pub/login.

Resolution

Route mapping reference

1. Confirm that the issue is a service discovery failure

• Command:

  curl -i --max-time 10 http://<gateway-host>:8200/portal/v4/pub/login

• Expected result: A parameter-validation or authentication response indicates that the route and target service are generally reachable.
• If abnormal: If the response is 503 and contains service-instance information, continue checking Nacos. If the TCP connection is refused or times out, troubleshoot the port and process first.
• Note: Gateway returns 404 for /actuator/**. Do not treat an Actuator 404 from Gateway as a Nacos failure.

2. Identify the service name for the failed path

• Check: Map the request path to the Nacos service name according to the Gateway configuration, and account for removal of the first segment by StripPrefix.
• Example: /portal/v4/pub/login should look up app-portal, and the target service receives /v4/pub/login.
• If abnormal: If the path prefix or service name does not match, correct the calling path or deployment configuration. Do not modify a Nacos instance to accommodate an incorrect path.

3. Check whether the target module service is running

• Commands:

  docker ps -a
  docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'

• Check: Identify the module container for the failed path, such as Portal for app-portal or the VM service for app-vm. Focus on container state and restart count.
• Expected result: The corresponding service container is Up or running stably, does not continuously restart, and listens on its expected port.
• If abnormal:
• If the container state is starting, wait for startup to complete and check again. If it remains starting, inspect its health-check configuration and startup logs.
• If the container state is exited, inspect its exit code and logs to determine whether a database, Redis, RabbitMQ, Nacos, configuration, or resource issue prevented startup.
• If the service is not running, remains starting, or is exited, do not continue modifying Nacos instances. Determine the specific startup failure, and contact support if necessary.
• Further commands:

  docker inspect <container-name>
  docker logs --since 2h <container-name>

4. Check effective Nacos settings for Gateway and the target service

• Check: Nacos address, username, password, namespace, group, service name, registered IP address, and port.
• Expected result: Gateway and the target service connect to the same Nacos server and use the same namespace and group. The target service name matches the route table.
• If abnormal: Pay particular attention to localhost in a multi-host or container deployment. localhost might refer to the current container instead of the Nacos host. Correct the runtime configuration and restart the affected service according to the deployment specification.

5. Query Nacos instances

• Read-only query example:

  curl -G --max-time 10 \
    'http://<nacos-host>:8848/nacos/v1/ns/instance/list' \
    --data-urlencode 'serviceName=app-portal' \
    --data-urlencode 'namespaceId=<effective-namespace>' \
    --data-urlencode 'groupName=DEFAULT_GROUP' \
    --data-urlencode 'healthy=true' \
    --data-urlencode 'enabled=true'

• Expected result: At least 1 healthy, enabled instance is returned, with an IP address and port reachable from the Gateway network.
• If abnormal: If the list is empty, check whether the service registered in the same namespace/group. If the instance is unhealthy, check target service startup logs and the Nacos heartbeat. If the API format is incompatible, perform an equivalent read-only query in the target Nacos console.

6. Test the target instance directly

• Command:

  curl -i --max-time 10 http://<instance-ip>:<instance-port>/v4/pub/login

• Expected result: The target instance returns an application-level response.
• If abnormal: If direct instance access fails, check the target service listen address, registered port, container network, and firewall. If direct access succeeds but Gateway still returns 503, return to checking the namespace, group, service name, and effective Gateway configuration.

7. Check logs and instance stability

• Recommended Gateway log search terms: Nacos, LoadBalancer, No servers available, no available service instance, Connection refused, and timeout.
• Recommended target service log search terms: register, registered, namespace, serviceName, startup failures, and port binding errors.
• Expected result: The target service completes registration, and Gateway no longer reports no available instance.
• If abnormal: If an instance disappears shortly after registration, check instance heartbeats, whether the service process repeatedly restarts, and health-check state.

8. Collect support information

• Failed request URL, HTTP status code, request time, and Gateway log.
• Effective Nacos configuration for Gateway and the target service, with passwords redacted.
• Nacos namespace, group, service name, instance IP address/port, and health/enabled state.
• Direct target-instance test result and service process/container state.
• Container name, image version, state, exit code, health-check result, and container logs from the failure window. Redact sensitive configuration before providing it to support.

Handling

1. Restore Nacos availability and authentication configuration so that Gateway and the target service can connect to Nacos.
2. Align the address, namespace, group, and service name used by Gateway and the target service.
3. Correct the registered IP address and port in Nacos so that they are reachable from the Gateway network.
4. If the target service is not running or is starting/exited, use the container logs to address dependency, configuration, resource, or health-check issues. If the cause cannot be confirmed, preserve the environment information and contact support.
5. After restoring the target service, confirm that it registers with Nacos again. Do not modify every Gateway route first.
6. Do not bypass Gateway service discovery by exposing every internal service port externally.

Verification

• Nacos: A healthy, enabled target service instance exists in the namespace/group used by Gateway.
• Service: docker ps -a shows that the target module service runs stably and is no longer starting, exited, or continuously restarting.
• Direct access: Each registered instance IP address and port returns an application-level response.
• Gateway: The original failed route no longer returns 503 and returns a normal business or parameter-validation response.
• Multiple routes: Verify at least 1 unaffected route and the original failed route.
• Logs: Messages indicating no available instance, a service discovery timeout, or a registration failure no longer appear during a complete request cycle.

Risks and rollback

• Changing a namespace, group, service name, or registered IP address affects service discovery. Record the original configuration first.
• Do not delete all instances in Nacos or bulk-modify other service registration information.
• If a configuration change fails, restore the original runtime configuration and restart only the affected services.

Additional information

• For the relationship between external port 8100 and Gateway port 8200, see How do I troubleshoot a ZCenter deployment or access failure caused by blocked ports?.
• Gateway is configured to return 404 for /actuator/**. Health checks should use the target service's direct address or the probe address explicitly configured in the target environment.