Troubleshooting intermittent "Cannot assign requested address" errors when a VM accesses object storage

Issue/Introduction

This article addresses a class of connection failures in which VMs running on ZStack Cloud access an external object storage endpoint and the application logs repeatedly report Cannot assign requested address.

The same pattern can affect any workload that frequently opens short-lived TCP connections from a Linux guest, such as an application server polling S3-compatible object storage.

Symptoms

The application log shows connection failures similar to the following:

2024-11-06 11:33:07 Unable to execute HTTP request: Connect to 169.169.32.177:8060 [/169.169.32.177] failed: Cannot assign requested address (connect failed)

The ZStack control plane, physical host, and network path are healthy. The error appears only in the guest OS of the VM and only under high concurrency.

Environment

• Product: ZStack Cloud; the issue originates from the guest OS of the VM rather than from the ZStack platform itself.
• Version: 4.x and 5.x.
• Component: Linux kernel network stack inside the guest OS.
• Deployment: high-frequency, short-lived outbound HTTP/HTTPS connections to a single destination such as object storage.

Possible causes

• The Linux guest has accumulated many sockets in TIME_WAIT.
• The ephemeral port range is too narrow for the workload's connection rate.
• The application repeatedly opens short-lived connections instead of using keep-alive or a persistent connection pool.

Resolution

1. Confirm ephemeral port exhaustion

1. Run netstat -ant | grep -c TIME_WAIT in the VM. If the count is high, for example thousands or more, and continues to increase, the VM may have exhausted its ephemeral port pool.
2. Run sysctl net.ipv4.ip_local_port_range in the VM. If the port range is narrow, for example the default 32768 60999 with about 28,000 ports, high-concurrency workloads may exhaust the range.
3. Confirm that the object storage service is reachable from another host with the same configuration. If it is reachable, the issue is local to the affected VM rather than the storage platform.

2. Apply one or more mitigation options

The following three measures are independent and can be combined as needed. Perform all operations in the guest OS of the affected VM Instance. Do not perform them on the management node or physical host.

Option 1 — Shorten the TIME_WAIT window and enable port reuse

sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.ipv4.tcp_timestamps=1
sysctl -w net.ipv4.tcp_tw_reuse=1

• tcp_fin_timeout=30 — shortens the TIME_WAIT socket release time from 60 seconds to 30 seconds.
• tcp_timestamps=1 — required for tcp_tw_reuse to take effect.
• tcp_tw_reuse=1 — allows TIME_WAIT sockets to be reused for new outbound connections.

Option 2 — Expand the ephemeral port range

sysctl -a | grep port_range
# Default: net.ipv4.ip_local_port_range = 32768 60999  (about 28,000 ports)

vi /etc/sysctl.conf
# Add or replace with:
net.ipv4.ip_local_port_range = 10000 65000  # about 55,000 ports

sysctl -p

No reboot is required.

Option 3 — Change short-lived connections to long-lived connections (keep-alive)

If the application protocol supports it, change the application to use long-lived connections, such as HTTP keep-alive or a persistent connection pool. This is the most sustainable fix because it directly removes the root cause instead of only increasing the port limit.

3. Verify the result

• After applying Option 1, run netstat -ant | grep -c TIME_WAIT again while the workload is running. The count should stabilize instead of continuously increasing.
• After applying Option 2, sysctl net.ipv4.ip_local_port_range should show the new port range.
• Under the same workload, the application error logs should no longer show new Cannot assign requested address errors.