Background
When performing stress testing on CLB, you may encounter connection failures caused by too many client TIME-WAIT (all ports are occupied in short time). Below are reasons and solutions:
Linux Parameter Introduction
tcp_timestamps: Determines whether the TCP timestamps option is enabled. Timestamps are negotiated during the TCP three-way handshake. If either party does not support it, the connection will not use the timestamps option.
tcp_tw_recycle: Determines whether the TCP TIME_WAIT state recycling is enabled.
tcp_tw_reuse: When enabled, connections in the TIME_WAIT state for more than 1 second can be directly recycled.
Cause Analysis
An excessive number of client TIME-WAIT states occur when the client actively disconnects. Each disconnected connection enters the TIME-WAIT state, with a default 60-second timeout for recycling. In general, clients encountering this scenario will enable the
tcp_tw_recycle and tcp_tw_reuse parameters to facilitate recycling connections in the TIME-WAIT state. However, the current CLB does not enable the tcp_timestamps option, rendering the client's tcp_tw_recycle and tcp_tw_reuse ineffective and unable to quickly recycle connections in the TIME-WAIT state. The following sections will explain the meanings of several Linux parameters and the reason why CLB cannot enable tcp_timestamps.1.
tcp_tw_recycle and tcp_tw_resuse only take effect when tcp_timestamps is enabled.2. In FullNAT scenarios, tcp_timestamps and tcp_tw_recycle cannot be enabled simultaneously, as public network clients accessing the server through the NAT gateway may encounter issues. The reasons are as follows:
When both tcp_tw_recycle and tcp_timestamps are enabled, the timestamp in the socket connect request from the same source IP host within 60 seconds must be incremental. Taking the 2.6.32 kernel as an example, the specific implementation is as follows:
if(tmp_opt.saw_tstamp && tcp_death_row.sysctl_tw_recycle &&(dst = inet_csk_route_req(sk,req))!= NULL &&(peer = rt_get_peer((struct rtable *)dst))!= NULL &&peer->v4daddr == saddr){if(get_seconds()< peer->tcp_ts_stamp + TCP_PAWS_MSL &&(s32)(peer->tcp_ts - req->ts_recent) > TCP_PAWS_WINDOW){NET_INC_STATS_BH(sock_net(sk),LINUX_MIB_PAWSPASSIVEREJECTED)；goto ↓drop_and_release；}}
Note:
tmp_opt.saw_tstamp: This socket supports tcp_timestamp.
sysctl_tw_recycle: The local system has the tcp_tw_recycle option enabled.
TCP_PAWS_MSL: 60s, this condition indicates that the last TCP communication from this source IP occurred within 60 seconds.
TCP_PAWS_WINDOW: 1, this condition indicates that the timestamp of the last TCP communication from this source IP is greater than the current TCP timestamp.
3. On CLB (Layer-7), tcp_timestamps is disabled because the public network client may fail to access the server through the NAT gateway, as shown in the example below:
3.1 A quintuple is still in the TIME_WAIT state. The NAT gateway's port allocation policy reuses the same quintuple within 2MSL and sends a SYN packet.
3.2 When tcp_timestamps is enabled, the SYN packet will be discarded if the following two conditions are met (since the timestamp option is enabled, it is considered an old packet):
3.2.1 Last timestamp > Current timestamp.
3.2.2 Within 24 days, packets have been received (the timestamp field is 32-bit, and Linux updates the timestamp every 1ms by default. Timestamp wraparound occurs after 24 days).
Note:
This issue is more prominent on mobile devices, as clients share limited public IP addresses under the carrier's NAT gateway. The five-tuple may be reused within the 2MSL period, and timestamps from different clients cannot be guaranteed to be incremental.
Taking 2.6.32 kernel as an example, the details are as follows:
static inline int tcp_paws_check(const struct tcp_options_received *rx_opt,int paws_win){if((s32)(rx_opt->ts_recent - rx_opt->rcv_tsval)<= paws_win)return 1;if(unlikely(get_seconds()>=rx_opt->ts_recent_stamp + TCP_PAWS_24DAYS))return 1;return 0;}
Note:
rx_opt->ts_recent: The timestamp of the previous instance.
rx_opt->rcv_tsval: The timestamp received this time.
get_seconds(): The current time.
rx_opt->ts_recent_stamp: The time when the last packet was received.
Solution
If the client has too many TIME_WAIT, see below for solutions:
With HTTP short connections (Connection: close), the CLB instance actively closes the connection, preventing the client from generating TIME_WAIT.
If the scenario requires using persistent connections, enable the SO_LINGER option on the socket and close the connection using RST to avoid entering the TIME_WAIT state, achieving the purpose of quickly recycling ports.