Diagnosing TIME_WAIT Socket Exhaustion — TCP Connection States and Kernel Tuning

What Is TIME_WAIT?

TIME_WAIT is the last state entered by the side that actively closed (active close) a TCP connection. Even after the connection is torn down, the socket is held for a period of time (typically 2×MSL; about 60 seconds on Linux). From the outside it can look puzzling—"why are so many already-finished connections still sitting around?"—but this is not a bug; it is intentional behavior for TCP reliability.

The problem shows up on high-load servers (especially reverse proxies and HTTP clients that open large numbers of short-lived connections) when TIME_WAIT sockets pile up into the tens or hundreds of thousands and ephemeral port exhaustion causes new outbound connections to fail. Errors such as cannot assign requested address are typical. This guide covers the causes and the correct responses.

The TCP State Machine and the 4-Way Handshake

Connection teardown proceeds via a 4-way handshake, and the side that sends FIN first (the active closer) goes through TIME_WAIT.

Active Close Passive Close
   ESTABLISHED                 ESTABLISHED
       | --- FIN ----------------> |
   FIN_WAIT_1                      |
       | <-- ACK ----------------- |
   FIN_WAIT_2                  CLOSE_WAIT
       | <-- FIN ----------------- |
       | --- ACK ----------------> | LAST_ACK
 TIME_WAIT ( 60 ) CLOSED
       |
   CLOSED

Why TIME_WAIT Exists

Why wait 60 seconds at all? There are two reasons.

1. Protection against loss of the final ACK: If the last ACK sent by the active closer is lost, the peer retransmits FIN. The socket must stay alive during TIME_WAIT so it can ACK that retransmitted FIN. Otherwise the peer receives an RST.
2. Preventing delayed old packets from mixing into a new connection: If a new connection is opened immediately with the same 4-tuple (src IP:port–dst IP:port), late packets from the previous connection still wandering the network can be delivered into the new connection. The 2×MSL wait guarantees those packets have all expired.

In other words, TIME_WAIT is a safety mechanism. Do not try to eliminate it blindly—start by looking at who is performing the active close.

Diagnosis: Checking State Distribution with ss

# socket
ss -s

#
# TCP:   180344 (estab 412, closed 178900, timewait 178500/0, ...)

If the timewait count exceeds tens of thousands, that is a warning sign. For a detailed breakdown:

#
ss -ant | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# TIME-WAIT socket
ss -tan state time-wait | head

# TIME-WAIT ( backend)
ss -tan state time-wait dst 10.0.0.20

ss is faster than netstat and is recommended. Seeing which destinations TIME_WAIT sockets concentrate on lets you infer who is doing the active close.

Ephemeral Port Exhaustion

Outbound connections borrow a source port one at a time from the ephemeral port range. If TIME_WAIT sockets toward the same destination occupy those ports, new connections cannot obtain a port.

# when check ( 32768~60999, 28k)
cat /proc/sys/net/ipv4/ip_local_port_range
sysctl net.ipv4.ip_local_port_range

#
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

The key here is 4-tuple uniqueness. If the destination IP:port is different, a new connection is possible even with the same source port. Therefore the most vulnerable case is making a large number of connections to a single backend. Review fundamental remedies in this order:

1. Keep-Alive / connection pooling: Reuse connections instead of opening a new one for every request (most effective)
2. Expand the ephemeral port range
3. Spread backends across multiple IPs
4. Kernel parameter tuning (below)

tcp_tw_reuse vs tcp_tw_recycle

This is where common bad advice needs to be corrected.

tcp_tw_recycle — Never Use It (Removed)

net.ipv4.tcp_tw_recycle used to reclaim TIME_WAIT sockets aggressively, but when multiple clients behind NAT appear as the same IP, timestamp-based decisions break and cause a serious bug where connections are randomly rejected. It was completely removed in Linux kernel 4.12. The parameter does not even exist on modern systems. If you see this value in old guides on the internet, ignore it.

tcp_tw_reuse — Conditionally Safe

net.ipv4.tcp_tw_reuse allows TIME_WAIT sockets whose safety has been confirmed via TCP timestamps to be reused for new connections, only for outbound (client-side) connections. Unlike recycle, it does not cause NAT problems.

# 1: loopback , 2: . 1 settings
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# reuse ( 1)
sysctl net.ipv4.tcp_timestamps

tcp_max_tw_buckets — The Upper Bound

# TIME_WAIT socket . when warning
sysctl net.ipv4.tcp_max_tw_buckets

Lowering the limit too aggressively breaks the TIME_WAIT safety functions described above, so be careful.

SO_REUSEADDR / SO_REUSEPORT

Related issues can also be addressed with socket options.

int opt = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

Most server frameworks enable SO_REUSEADDR by default, which prevents bind failures on restart.

Recommended Application Procedure

# /etc/sysctl.d/99-tcp-tuning.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
# net.ipv4.tcp_tw_recycle ← /

#
sudo sysctl --system

Summary

TIME_WAIT is not an enemy to be eliminated but a safety mechanism to understand and work with. If you first look at "who is doing the active close" and "whether connections are being reused," most exhaustion problems will be solved.