nginx SYN flooding解决思路

问题描述

业务最近发现 nginx 反向代理服务器SYN 连接特别多

lf-weather-nginx-101-11:~ # tail -f /var/log/messages

Dec 24 09:31:29 lf-weather-lvs-101-11 kernel: [7406388.265092] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:32:50 lf-weather-lvs-101-11 kernel: [7406469.537620] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:35:02 lf-weather-lvs-101-11 kernel: [7406600.783795] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:38:30 lf-weather-lvs-101-11 kernel: [7406809.145161] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:39:57 lf-weather-lvs-101-11 kernel: [7406895.519336] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:41:05 lf-weather-lvs-101-11 kernel: [7406964.278512] possible SYN flooding on port 8080. Sending cookies.

Dec 24 09:44:26 lf-weather-lvs-101-11 kernel: [7407164.819061] possible SYN flooding on port 8080. Sending cookies.

解决步骤

1.网络TCP UDP连接排名

lf-weather-nginx-101-11:~ # netstat -ntu | awk '{print $5"\n"}' | cut -d: -f1 | sort | uniq -c | sort -nr|head -n 20

54456

44 220.202.103.116

41 61.50.248.5

37 111.20.241.255

30 220.201.8.217

29 113.57.255.1

28 61.55.156.20

24 221.226.97.19

22 10.51.241.104

20 121.31.254.44

17 61.158.153.214

17 122.96.104.19

14 61.158.152.232

14 61.158.152.215

14 61.158.152.202

14 202.99.82.23

14 101.254.209.16

13 61.158.152.241

13 61.158.152.225

13 61.158.152.192

正常

2.连接状态排名

lf-weather-nginx-101-11:~ # netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

TIME_WAIT 1293

CLOSE_WAIT 1

FIN_WAIT1 3428

ESTABLISHED 5330

FIN_WAIT2 46193

SYN_RECV 371

CLOSING 32

LAST_ACK 113

发现FIN_WAIT2特别多不正常

3.nginx client IP 连接数排名

lf-weather-nginx-101-12:/opt/huawei/nginx/logs # awk '{print $1}' access__xx.log|awk '{count[$1]++}END{for(i in count)print i,count[i]}'|sort -n -k 2 -r |head -n 10

112.97.30.1 417

61.50.248.5 371

61.55.156.20 368

119.145.15.27 361

58.251.152.27 351

61.158.153.164 310

220.202.103.116 287

61.158.153.214 279

221.226.97.19 218

220.201.8.217 216

4.调整后的内核差数如下

lf-weather-nginx-101-12: # cat /etc/sysctl.conf

# Disable response to broadcasts.

# You don't want yourself becoming a Smurf amplifier.

net.ipv4.icmp_echo_ignore_broadcasts = 1

# enable route verification on all interfaces

net.ipv4.conf.all.rp_filter = 1

# enable ipV6 forwarding

#net.ipv6.conf.all.forwarding = 1

# increase the number of possible inotify(7) watches

fs.inotify.max_user_watches = 65536

# avoid deleting secondary IPs on deleting the primary IP

net.ipv4.conf.default.promote_secondaries = 1

net.ipv4.conf.all.promote_secondaries = 1

net.ipv4.ip_forward=0

net.ipv4.conf.default.arp_filter=0

net.ipv4.conf.default.accept_redirects=0

net.ipv4.conf.all.accept_redirects=0

net.ipv4.conf.default.send_redirects=0

net.ipv4.conf.all.send_redirects=0

net.ipv4.conf.all.secure_redirects=0

net.ipv4.conf.default.secure_redirects=0

net.ipv4.conf.default.rp_filter=1

net.ipv4.conf.all.log_martians=1

net.ipv4.conf.default.log_martians=1

net.ipv4.tcp_timestamps=0

net.ipv4.icmp_ignore_bogus_error_responses=1

net.ipv4.conf.all.proxy_arp=0

net.ipv4.conf.default.proxy_arp=0

kernel.sysrq=8

net.ipv4.tcp_syncookies=1 #表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭

net.ipv4.conf.default.accept_source_route=0

net.ipv4.conf.all.accept_source_route=0

net.ipv4.route.gc_interval=1

net.ipv4.route.gc_timeout=30 #tcp连接超时时间 之前是60

net.ipv4.ip_local_port_range = 1024 65535 #tcp端口的使用范围

net.ipv4.tcp_tw_reuse = 1 #表示开启TCP连接中TIME-WAIT sockets的快速回收 新添加

net.ipv4.tcp_tw_recycle = 1 #和net.ipv4.tcp_tw_reuse 同时开启才能生效

net.core.somaxconn = 262144 #web应用中listen函数的backlog默认会将内核参数的net.core.somaxconn限制到128，而nginx定义的NGX_LISTEN_BACKLOG默认是511，所以必须调整

net.ipv4.tcp_fin_timeout = 30 #表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。

net.ipv4.tcp_keepalive_probes = 3

net.ipv4.tcp_keepalive_intvl = 10

net.ipv4.tcp_max_syn_backlog = 262144 #表示SYN队列的长度，默认为1024，加大队列长度为262144，可以容纳更多等待连接的网络连接数。

net.ipv4.tcp_max_tw_buckets = 262144 #表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息

net.core.netdev_max_backlog = 262144 #每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目

lf-weather-nginx-101-12: #sysctl -p

几分钟后看FIN_WAIT2明显减少

lf-weather-nginx-101-12:/opt/huawei/nginx/logs # ss -s

Total: 4540 (kernel 4691)

TCP: 33564 (estab 4463, closed 25651, orphaned 3428, synrecv 0, timewait 25651/0), ports 355

Transport Total IP IPv6

* 4691 - -

RAW 0 0 0

UDP 7 7 0

TCP 7913 7913 0

INET 7920 7920 0

FRAG 0 0 0

再看日志 SYN依然存在 不过一分钟出现一个还能接受吧 排除攻击业务访问也正常

lf-weather-nginx-101-12:/opt/huawei/nginx/logs # tail -f /var/log/messages

Dec 24 10:15:01 lf-weather-lvs-101-12 kernel: [7408800.117536] possible SYN flooding on port 8080. Sending cookies.

Dec 24 10:16:03 lf-weather-lvs-101-12 kernel: [7408861.984733] possible SYN flooding on port 8080. Sending cookies.

Dec 24 10:19:51 lf-weather-lvs-101-12 kernel: [7409089.678098] possible SYN flooding on port 8080. Sending cookies.

Dec 24 10:21:41 lf-weather-lvs-101-12 kernel: [7409199.529564] possible SYN flooding on port 8080. Sending cookies.

Dec 24 10:23:08 lf-weather-lvs-101-12 kernel: [7409286.745773] possible SYN flooding on port 8080. Sending cookies.