F5＋Nginx基于IP实现负载均衡和session保持

F5 --> Nginx --> App01,App02

Nginx获得的是F5 IP,无法通过IP HASH实现load balance.

但F5通过X-Forward-For头送了真实IP,于是模块real_ip可以派上用场，和IP HASH配合。

加入real ip模块

./configure --with-debug --with-cc-opt='-O0' --with-http_realip_module --prefix=/opt/nginx/nginx136

配置nginx.conf

log_format main '$remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for" "$realip_remote_addr"';

set_real_ip_from 192.168.0.252;

real_ip_header X-Forwarded-For;

real_ip_recursive on;

upstream cgbups {

ip_hash;

server 192.168.0.252:8082;

server 192.168.0.252:8083;

}

server {

listen 8080;

server_name localhost;

location / {

index index.html index.htm;

proxy_pass http://cgbups;

}

}

模拟没有XFF头的请求，

wgethttp://192.168.0.252:8080/ok.html

看到access log如下,

192.168.0.252 - - [13/Oct/2018:22:50:24 +0800] "GET /ok.html HTTP/1.1" 200 2 "-" "Wget/1.16.3 (linux-gnu)" "-" "192.168.0.252"

#模拟F5发XFF头

wget --header="X-Forwarded-For: 21.96.88.10" http://192.168.0.252:8080/ok.html

看到access log如下,注意第一个字段的remote_address已经变为XFF送来的IP了

值得注意的是Nginx IP Hash模块拿的是C类地址做hash.

case AF_INET:

sin = (struct sockaddr_in *) r->connection->sockaddr;

iphp->addr = (u_char *) &sin->sin_addr.s_addr;

iphp->addrlen = 3;

这个过程可以在代码里加一行debug输出更直接

ngx_http_upstream_get_ip_hash_peer(ngx_peer_connection_t *pc, void *data)

{

ngx_http_upstream_ip_hash_peer_data_t *iphp = data;

......

hash = iphp->hash;

ngx_log_debug3(NGX_LOG_DEBUG_HTTP, pc->log, 0,

"real ip %ui.%ui.%ui", iphp->addr[0],iphp->addr[1],iphp->addr[2]);

for ( ;; ) {

然后把error 改为debug级别，

就可以在error log看到加了XFF和不加的区别。

2018/10/13 22:50:24 [debug] 26917#0: *7 real ip 192.168.0

2018/10/13 22:52:08 [debug] 26917#0: *9 real ip 21.96.94

2018/10/13 22:52:22 [debug] 26917#0: *11 real ip 21.96.95

用gdb跟踪打印同样方便。

(gdb) p iphp->addr[0]

$5 = 21 '\025'

(gdb) p iphp->addr[1]

$6 = 96 '`'

(gdb) p iphp->addr[2]

$7 = 95 '['