nginx一般直接在配置文件里配置upstream即可实现负载均衡,但有些特定的环境下此种方式就显得有些局限性。比如后端服务器无法依据端口占用检查存活的时候;后台动态调整节点的时候;调整节点后不想修改配置文件重启nginx的时候等等。
此文的思路是将配置文件从nginx本地迁移到其他第三方服务上如etcd、consul上,然后时候拉取配置到本地。理论上说任何第三方配置中心都可以实现该功能,但需要对应的nginx模块。本文采用nginx-upsync-module,主要支持consul、etcd,本文以consul为例。
迁移配置文件还无法满足需求,还需要解决服务检测机制。这里不再以端口占用为准,而是实际访问某一个接口,查看是否返回数据,并以此为存活依据,最后通过调用consul的rest接口管理配置。
nginx安装参照前文《Linux下Nginx1.8安装》
需要注意的是,在安装nginx的时候需要安装nginx-upsync-module模块。
打开https://github.com/weibocom/nginx-upsync-module,如果遇到github打不开,可以参照如下链接解决:https://www.php.cn/faq/445082.html
下载完成后,解压到linux目录备用。
consul的安装比较简单,这里不再赘述,可参照如下链接:https://blog.csdn.net/junaozun/article/details/90699384
本文的基础目录为:
1、nginx源码目录:/opt/server/software/nginx-1.19.1
2、nginx安装目录:/home/nginx/nginx
3、nginx-upsync-module目录:/opt/server/software/nginx-upsync-module/
确定consul正常运行后,配置nginx:
consul.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream testconsul {
#这个配置无用了,但删除后启动会报错,如果consul里没有配置则会调用此地址,故此设置应该设置为一个默认的可用的地址,一旦从consul拉取到数据这个配置就无用了
server 127.0.0.1:11111;
#### 连接consul server,获取动态upstreams,配置负载均衡信息,间隔0.5s获取配置信息,upsync_timeout配置从consul拉取上游服务器配置的超时时间;upsync_interval配置从consul拉取上游服务器配置的间隔时间;upsync_type指定使用consul配置服务器;strong_dependency配置nginx在启动时是否强制依赖配置服务器,如果配置为on,则拉取配置失败时nginx启动同样失败
upsync 192.168.1.97:8500/v1/kv/upstreams/testconsul/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
### 动态获取consul server相关负载均衡配置信息持久化在硬盘,这样即使consul服务器出问题了,本地还有一个备份。
upsync_dump_path /home/nginx/nginx/conf/servers/testconsul.conf;
}
server {
listen 80 ;
server_name testconsul;
charset utf8;
location /{
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://testconsul/HelloWord;
}
access_log /home/nginx/nginx/logs/access-upsync.log;
location /stub-status {
stub_status on;
}
location = /upstream_show {
upstream_show;
}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
}
URL目录
1、负载状态:http://nginxhost:port/stub-status
2、当前负载节点列表:http://nginxhost:port/upstream_show
3、测试URL:http://nginxhost:port/HelloWord
4、consul增加节点(也可直接在UI上添加):http://consulhost:port/v1/kv/upstreams/consultest/192.168.1.22:8080
核心功能是根据配置检测服务是否可用,然后根据结果更新到consul上。代码比较简单,主要代码如下:
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.cmgplex.aotucheck4consul</groupId>
<artifactId>aotucheck4consul</artifactId>
<version>1.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>aotucheck4consul</name>
<url>http://www.cmgplex.com</url>
<parent>
<groupId>com.cmgplex.hr.parent</groupId>
<artifactId>hr-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath></relativePath>
</parent>
<properties>
<snippetsDirectory>${project.build.directory}/generated-snippets</snippetsDirectory>
<asciidoctor.input.directory>${project.basedir}/docs/asciidoc</asciidoctor.input.directory>
<generated.asciidoc.directory>${project.build.directory}/asciidoc</generated.asciidoc.directory>
<asciidoctor.html.output.directory>${project.build.directory}/asciidoc/html</asciidoctor.html.output.directory>
<asciidoctor.pdf.output.directory>${project.build.directory}/asciidoc/pdf</asciidoctor.pdf.output.directory>
</properties>
<dependencies>
<!--base dependency for all start -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>net.logstash.log4j</groupId>
<artifactId>jsonevent-layout</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</dependency>
<!--base dependency for all end -->
<!--jetty begin -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-undertow</artifactId>
</dependency>
<!--jetty end -->
<!--consul begin -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
<!--consul end -->
<!--stream rabbit begin -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
<!--stream rabbit end -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>
<!--feign begin -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<!--feign end -->
<!--actuator begin -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!--actuator end -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.46</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4.7</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
</dependency>
<!-- consul api begin -->
<dependency>
<groupId>com.ecwid.consul</groupId>
<artifactId>consul-api</artifactId>
<version>1.4.5</version>
</dependency>
<!-- consul api end -->
</dependencies>
<build>
<finalName>aotucheck4consul</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.xml</include>
</includes>
</resource>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.*</include>
</includes>
</resource>
<resource>
<directory>src/main/webapp</directory>
<targetPath>META-INF/resources</targetPath>
<includes>
<include>**/*.*</include>
</includes>
</resource>
</resources>
</build>
<!-- 这里是为了方便未配置maven的情况下,找到依赖 -->
<repositories>
<repository>
<id>lumi-snapshots</id>
<name>lumi-snapshots</name>
<url>http://repo.lumiai.top/repository/maven-snapshots/</url>
<layout>default</layout>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>
application.properties
spring.main.allow-bean-definition-overriding=true
#the following is for cloud
spring.datasource.hikari.idle-timeout = 600000
spring.datasource.hikari.connection-timeout = 30000
spring.datasource.hikari.max-lifetime = 1800000
spring.datasource.hikari.maximum-pool-size = 20
spring.datasource.type = com.zaxxer.hikari.HikariDataSource
#这个是关于mq的配置
spring.rabbitmq.addresses=amqp://192.168.1.2:5672
spring.rabbitmq.password=guest
spring.rabbitmq.username=guest
spring.cloud.stream.default-binder=rabbit
#定时检查,每隔1分钟
job.cleanConsulDeadService.cron=1 * * * * ?
#consul配置key默认前缀
consul.default.key.prefix=upstreams/
#服务检查配置
#----------------api服务
#要检查服务的url,不包含地址
server.check.apps.api.url=/api/b
#要检查服务的host,只能是IP,包含端口,即使是80也不能少,不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.api.hosts=http://192.168.1.1:80,http://192.168.1.2:81
#要检查服务的method
server.check.apps.api.method=get
#暂不支持json参数
server.check.apps.api.param=get
#链接超时时间,单位毫秒
server.check.apps.api.timeout=3000
#断定服务正常的response返回code
server.check.apps.api.successcode=200,302
#----------------baidu服务
#要检查服务的url,不包含地址
server.check.apps.consultest.url=/s
#要检查服务的host,只能是IP,包含端口,即使是80也不能少,不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.consultest.hosts=http://182.61.200.6:80
#要检查服务的method
server.check.apps.consultest.method=get
#暂不支持json参数
server.check.apps.consultest.param=wd=javahttp&tn=98012088_5_dg&ch=11
#链接超时时间,单位毫秒
server.check.apps.consultest.timeout=3000
#断定服务正常的response返回code
server.check.apps.consultest.successcode=200,302
#********************************************
#本地服务监听端口
server.port = 17000
#cpu的核数
server.undertow.io-threads=4
#预估的最佳线程数
server.undertow.worker-threads=400
spring.application.name=aotucheck4consul
spring.profiles.active=dev
#********************************************
#consul的配置
spring.cloud.consul.enabled=true
spring.cloud.consul.host=192.168.1.2
spring.cloud.consul.port=8500
#********************************************
#注册中心的配置
spring.cloud.consul.discovery.enabled=false
spring.cloud.consul.discovery.prefer-ip-address=true
spring.cloud.consul.discovery.health-check-path=/actuator/health
spring.cloud.consul.discovery.health-check-interval=10s
spring.cloud.consul.discovery.health-check-timeout=1s
spring.cloud.consul.discovery.instance-id=${spring.application.name}:${spring.cloud.client.ip-address}:${server.port}
#是否把自己向注册中心注册,可以做纯consumer不注册自己
spring.cloud.consul.discovery.register=true
#********************************************
#配置服务的配置
spring.cloud.consul.config.enabled=true
spring.cloud.consul.config.prefix=config
spring.cloud.consul.config.profile-separator=,
spring.cloud.consul.config.default-context=application
spring.cloud.consul.config.watch.enabled=true
spring.cloud.consul.config.watch.delay=1000
spring.cloud.consul.config.watch.wait-time=3
spring.cloud.consul.config.format=properties
spring.cloud.consul.config.data-key=configuration
#禁用熔断器首次调用时强制1秒超时
hystrix.command.default.execution.timeout.enabled=false
CheckServiceScheduler.java
@Component
@EnableScheduling
public class CheckServiceScheduler {
private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceScheduler.class);
@Autowired
private ICheckService checkService;
@Scheduled(cron = "${job.cleanConsulDeadService.cron}")
public void checkServer() {
LOGGER.info("-----------------------------start checkServer Scheduler-----------------------------");
this.checkService.check();
LOGGER.info("-----------------------------end checkServer Scheduler-----------------------------");
}
}
CheckServiceImpl.java
@Service
public class CheckServiceImpl implements ICheckService {
private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceImpl.class);
@Resource
private ServerPropertiesConfig serverPropertiesConfig;
@Resource
private IConsulService consulService;
@Value("${consul.default.key.prefix}")
private String consulKeyPrefix;
/*
* (非 Javadoc) <p>Title: check</p> <p>Description: </p>
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.ICheckService#check()
*/
@Override
public void check() {
LOGGER.debug("serverPropertiesConfig={}", this.serverPropertiesConfig);
Map<String, ServerInfoVo> apps = this.serverPropertiesConfig.getApps();
for (Entry<String, ServerInfoVo> app : apps.entrySet()) {
String appName = app.getKey();
LOGGER.info("appName={}", appName);
ServerInfoVo config = app.getValue();
LOGGER.info("config={}", config);
List<Integer> expectCodeList = ListUtils.str2list(config.getSuccesscode(), ",");
List<String> hostsList = ListUtils.str2list4String(config.getHosts(), ",");
String method = config.getMethod();
HttpMethod httpMethod = HttpMethod.valueOf(method.toUpperCase());
String param = config.getParam();
Integer timeout = config.getTimeout();
hostsList.parallelStream().forEach(host -> {
try {
LOGGER.info("host={}", host);
String httpurl = host + config.getUrl();
LOGGER.info("httpurl={}", httpurl);
boolean isSuccess = HttpTools.compareCode(expectCodeList, httpMethod, httpurl, param, timeout);
LOGGER.info("isSuccess={}", isSuccess);
String key = this.consulKeyPrefix + appName + "/"
+ host.replace("http://", "").replace("https://", "");
LOGGER.info("key={}", key);
// 如果服务可用
if (isSuccess) {
// 检查consul上配置是否正常
if (!this.consulService.isKeyExist(key)) {
// 如果不正常,则更新正常
this.consulService.addKeyValue(key, new NginxServerConfig(2, 1, 10));
LOGGER.info("server success,consul faild,now update consul,key={}", key);
}
} else {
// 如果服务不可用
// 检查consul上配置是否正常
if (this.consulService.isKeyExist(key)) {
// 如果正常,则更新为不正常
this.consulService.deleteKey(key);
LOGGER.info("server faild,consul success,now update consul,key={}", key);
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
});
}
}
}
ConsulServiceImpl.java
@Service
public class ConsulServiceImpl implements IConsulService {
private static final Logger LOGGER = LoggerFactory.getLogger(ConsulServiceImpl.class);
private static final BASE64Decoder decoder = new BASE64Decoder();
@Autowired
private ConsulClient consulClient;
/*
* (非 Javadoc) <p>Title: getKeysByPrefix</p> <p>Description: </p>
*
* @param keyPrefix
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getKeysByPrefix(java.lang.String)
*/
@Override
public List<String> getKeysByPrefix(String keyPrefix) {
Response<List<GetValue>> valus = this.consulClient.getKVValues(keyPrefix);
return valus.getValue().stream().map(value -> value.getKey()).filter(key -> StringUtils.isNotBlank(key))
.collect(Collectors.toList());
}
/*
* (非 Javadoc) <p>Title: getValue</p> <p>Description: </p>
*
* @param key
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getValue(java.lang.String)
*/
@Override
public NginxServerConfig getValue(String key) {
GetValue kvValue = this.consulClient.getKVValue(key).getValue();
if (null == kvValue) {
return null;
}
String value = kvValue.getValue();
String jsonValue;
try {
jsonValue = new String(decoder.decodeBuffer(value), "UTF-8");
return JSON.parseObject(jsonValue, NginxServerConfig.class);
} catch (IOException e) {
LOGGER.error(e.getMessage(), e);
}
return null;
}
/*
* (非 Javadoc) <p>Title: isKeyExist</p> <p>Description: </p>
*
* @param key
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#isKeyExist(java.lang.String)
*/
@Override
public boolean isKeyExist(String key) {
return null != this.getValue(key);
}
/*
* (非 Javadoc) <p>Title: deleteKey</p> <p>Description: </p>
*
* @param key
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#deleteKey(java.lang.String)
*/
@Override
public boolean deleteKey(String key) {
return null != this.consulClient.deleteKVValue(key);
}
/*
* (非 Javadoc) <p>Title: addKeyValue</p> <p>Description: </p>
*
* @param key
*
* @param value
*
* @return
*
* @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#addKeyValue(java.lang.String,
* com.cmgplex.aotucheck4consul.jobsystem.vo.NginxServerConfig)
*/
@Override
public boolean addKeyValue(String key, NginxServerConfig value) {
return this.consulClient.setKVValue(key, JSON.toJSONString(value)).getValue();
}
}