Fixing Linux Ephemeral Port Exhaustion and TIME_WAIT Drops
calricon / field note
Title: Fixing Linux Ephemeral Port Exhaustion and TIME_WAIT Drops Slug: fixing-linux-ephemeral-port-exhaustion-time-wait-drops Focus Keyword: cannot assign requested address eaddrnotavail time_wait Meta Title: Fix Linux Ephemeral Port Exhaustion &...
Fixing Linux Ephemeral Port Exhaustion and TIME_WAIT Drops
High-concurrency Linux reverse proxies, API gateways, and VoIP/Voice AI worker nodes frequently begin failing outbound connections with connect() failed (99: Cannot assign requested address) (EADDRNOTAVAIL) during traffic bursts—even while host CPU, RAM, and NIC bandwidth remain under 40% utilization.
This failure occurs when the kernel exhausts its outbound ephemeral TCP port pool. Linux's default ip_local_port_range (32768–60999) provides only ~28,232 ephemeral ports per unique destination tuple. When short-lived HTTP or database connections close without pooling, sockets sit in TIME_WAIT for 60 seconds (2 * MSL), starving new outbound connect() calls until ports age out.
Problem Diagnosis & Root Cause Architecture
Eliminating EADDRNOTAVAIL socket exhaustion requires understanding how Linux allocates 4-tuple ports and recycles TIME_WAIT entries:
- Narrow Default Ephemeral Port Range: At 500+ short-lived outbound requests per second to a single backend IP and port, 28,232 available ports saturate in under 57 seconds as closed sockets accumulate in
TIME_WAIT. - Missing Persistent Connection Pooling: Opening a fresh TCP/TLS connection per request (instead of reusing HTTP/1.1 keep-alive or HTTP/2 multiplexed sockets) multiplies
TIME_WAITchurn linearly with request rate. - Disabled
tcp_tw_reuseon Outbound Clients: Withnet.ipv4.tcp_tw_reuse = 0(default), Linux refuses to reclaimTIME_WAITsockets for new outboundconnect()calls even when TCP timestamps prove the old flow has drained. - Obsolete
tcp_tw_recycleDanger: Legacy tuning guides often recommendnet.ipv4.tcp_tw_recycle = 1, which was removed in Linux 4.12 because it silently drops incoming connections from clients sharing a NAT gateway.
Never attempt to setnet.ipv4.tcp_tw_recycleon modern Linux kernels—it was removed in kernel 4.12 due to severe NAT packet drops. Usenet.ipv4.tcp_tw_reuse = 1(or2for loopback-only on newer kernels) alongsidetcp_timestamps = 1for safe outbound socket reuse.
Why Standard Implementations Fail
1. Tuning Kernel Sysctls Without Enabling Upstream Keep-Alive
Expanding ephemeral ports from 28,000 to 64,000 buys only an extra ~60 seconds of headroom if NGINX, Node.js, or Python workers still open and close a new TCP socket for every request.
2. Disabling tcp_timestamps While Enabling tcp_tw_reuse
tcp_tw_reuse relies on TCP PAWS timestamps to ensure old packets cannot corrupt a reused port; setting net.ipv4.tcp_timestamps = 0 renders tcp_tw_reuse ineffective.
3. Leaving tcp_max_tw_buckets Undersized
If TIME_WAIT sockets exceed tcp_max_tw_buckets, the kernel logs TCP: time wait bucket table overflow and abruptly resets connections.
Validated Step-by-Step Resolution Playbook
Step 1: Measure Live TIME_WAIT Counts and Port Range
Inspect socket states grouped by status, check the active ephemeral port range, and verify current sysctl flags:
ss -s
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -nr
sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse net.ipv4.tcp_timestamps net.ipv4.tcp_fin_timeoutStep 2: Expand Ephemeral Ports and Enable Safe TIME_WAIT Reuse
Create /etc/sysctl.d/99-tcp-ephemeral.conf to widen the local port range to 1024 65535 (ensuring no service on the host binds static ports inside that range, or use 10000 65535 if local services listen below 10000), enable tcp_tw_reuse, and shorten tcp_fin_timeout:
cat <<'EOF' > /etc/sysctl.d/99-tcp-ephemeral.conf
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 262144
EOF
sysctl --systemStep 3: Enforce Persistent Connection Pooling in Proxies & Workers
In NGINX upstreams, configure keepalive 64; with proxy_http_version 1.1; and proxy_set_header Connection "";. In Python/Node/Go microservices, share a single module-level HTTP client session across requests rather than creating a new client per call.
Step 4: Production Verification
Confirm the applied sysctl values, monitor TIME_WAIT socket counts under peak traffic, and verify zero EADDRNOTAVAIL errors in proxy logs:
sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse
ss -tan state time-wait | wc -l
grep -i "Cannot assign requested address" /var/log/nginx/error.log | tail -n 10Combining an expanded ephemeral port range, safe tcp_tw_reuse, and persistent HTTP keep-alive eliminates outbound socket exhaustion on high-concurrency Linux nodes.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.