grafana+prometheus+Consul自动服务发现监控平台

tar -zxvf prometheus-2.23.0.linux-amd64.tar.gz
vi /gpe/prometheus-2.23.0.linux-amd64/prometheus.yml
cd prometheus-2.23.0.linux-amd64
nohup ./prometheus --config.file=prometheus.yml >/dev/null &

可以通过http://ip:9090访问浏览器，或者http://ip:9090/metrics看是否能提供关于自身的各项指标

tar -zxvf grafana-7.3.4.linux-amd64.tar.gz
cd grafana-7.3.4
nohup ./bin/grafana-server web >/dev/null &

访问浏览器http://ip:3000, 默认的账号密码为: admin / admin,登录后会要求修改密码

[smtp]
enabled = true
host = smtp.163.com:465  #smtp服务器的地址：端口 (服务器地址不同公司可能不相同)
user = 你的邮箱
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
password = 邮箱客户端授权密码
;cert_file =
;key_file =
skip_verify = true  #Verify SSL for smtp server? defaults to false
from_address = 你的邮箱
from_name = Grafana

docker run -d \
        --name node-exporter \
        -p 9100:9100 \
        -v "/proc:/host/proc" \
        -v "/sys:/host/sys" \
        -v "/:/rootfs" \
        --net="host" \
        prom/node-exporter

- job_name: 'node-90.19'
    static_configs:
    - targets: ['172.16.90.19:9100']
      labels:
        env: test
        name: server-90.19
        instance: 172.16.90.19

docker run -d \
        --name mysqld-exporter \
        -p 9104:9104 \
        -e DATA_SOURCE_NAME="root:123456@(172.16.90.9:3306)/nacos-test" \
        prom/mysqld-exporter

- job_name: 'mysqld'
    static_configs:
    - targets: ['172.16.90.34:9104']
      labels:
        env: test
        name: mysql-5.7
        instance: 172.16.90.9:3306

docker run -d \
       --name redis_exporter \
       -p 9121:9121 \
       oliver006/redis_exporter \
       --redis.addr redis://172.16.90.9:6379 \
       --redis.password 123456

- job_name: redis
    static_configs:
    - targets: ['172.16.90.34:9121']
      labels:
        env: test
        name: redis
        instance: 172.16.90.34:6379

mkdir -pv /etc/consul.d/  && mkdir -pv /data/consul/ && mkdir -pv /data/consul/shell

[root@localhost consul.d]# cat /etc/consul.d/server.json 
{
  "datacenter": "dc1",
  "data_dir": "/data/consul",
  "log_level": "INFO",
  "node_name": "consul-server",
  "server": true,
  "bootstrap_expect": 1,
  "bind_addr": "172.16.90.19",
  "client_addr": "172.16.90.19",
  "ui":true,
  "retry_join": ["172.16.90.19"],
  "retry_interval": "10s",
  "enable_debug": false,
  "rejoin_after_leave": true,
  "start_join": ["172.16.90.19"],
  "enable_syslog": true,
  "syslog_facility": "local0"
}

启动consul server 启动服务
nohup consul agent -config-dir=/etc/consul.d > /data/consul/consul.log & 

访问浏览器http://ip:8500

curl -X PUT -d '{
"id": "node-exporter",
"name": "node-exporter-19.19",
"address": "172.16.90.19",
"port": 9100,
"tags": ["test"],
"checks": [{"http": "http://172.16.90.19:9100/metrics", "interval": "5s"}]}'  
http://172.16.90.19:8500/v1/agent/service/register

### 参数说明
# id : 注册ID 在consul中为唯一标识
# name ：Service名称
# address：自动注册绑定ip
# port：自动注册绑定端口
# tags：注册标签，可多个
# checks : 健康检查
#   http:   检查数据来源
#   interval: 检查时间间隔
# http://192.168.0.181:8500/v1/agent/service/register   consul注册接口

curl -X PUT -d '{"id": "node-exporter","name": "node-exporter-19.19","address": "172.16.90.19","port": 9100,"tags": ["test"],"checks": [{"http": "http://172.16.90.19:9100/metrics", "interval": "5s"}]}'  http://172.16.90.19:8500/v1/agent/service/register

如果要注销掉某个服务，可以通过如下 API 命令操作，例如注销上边添加的 node-exporter 服务

curl -X PUT http://172.16.90.19:8500/v1/agent/service/deregister/node-exporte

现在 Consul 服务已经启动完毕，并成功注册了一个服务，接下来，我们需要配置 Prometheus 来使用 Consul 自动服务发现，目的就是能够将上边添加的服务自动发现到 Prometheus 的 Targets 中，增加 prometheus.yml 配置如下：
  - job_name: 'consul-prometheus'
    consul_sd_configs:
    - server: '172.16.90.19:8500'
      services: []  

我们可以配置 relabel_configs 来实现标签过滤，只加载符合规则的服务。以上边为例，可以通过过滤 __meta_consul_tags 标签为 test 的服务，relabel_config 向 Consul 注册服务的时候，只加载匹配 regex 表达式的标签的服务到自己的配置文件。修改 prometheus.yml 配置如下：

  - job_name: 'consul-prometheus'
    consul_sd_configs:
    - server: '172.16.90.19:8500'
      services: []  
    relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex: .*test.*
      action: keep