使用prometheus operator监控envoy
文章目录
kubernetes集群三步安装
概述
prometheus operator应当是使用监控系统的最佳实践了，首先它一键构建整个监控系统，通过一些无侵入的手段去配置如监控数据源等 故障自动恢复，高可用的告警等。。
不过对于新手使用上还是有一丢丢小门槛，本文就结合如何给envoy做监控这个例子来分享使用prometheus operator的正确姿势
至于如何写告警规则，如何配置prometheus查询语句不是本文探讨的重点，会在后续文章中给大家分享，本文着重探讨如何使用prometheus operator
prometheus operator安装
sealyun离线安装包内已经包含prometheus operator,安装完直接使用即可
配置监控数据源
原理：通过operator的CRD发现监控数据源service
启动envoy
apiVersion: apps/v1
kind: Deployment
metadata:
name: envoy
labels:
app: envoy
spec:
replicas: 1
selector:
matchLabels:
app: envoy
template:
metadata:
labels:
app: envoy
spec:
volumes:
- hostPath: # 为了配置方便把envory配置文件挂载出来了
path: /root/envoy
type: DirectoryOrCreate
name: envoy
containers:
- name: envoy
volumeMounts:
- mountPath: /etc/envoy
name: envoy
readOnly: true
image: envoyproxy/envoy:latest
ports:
- containerPort: 10000 # 数据端口
- containerPort: 9901 # 管理端口，metric是通过此端口暴露
---
kind: Service
apiVersion: v1
metadata:
name: envoy
labels:
app: envoy # 给service贴上标签，operator会去找这个service
spec:
selector:
app: envoy
ports:
- protocol: TCP
port: 80
targetPort: 10000
name: user
- protocol: TCP # service暴露metric的端口
port: 81
targetPort: 9901
name: metrics # 名字很重要，ServiceMonitor 会找端口名envoy配置文件： 监听的地址一定需要修改成0.0.0.0，否则通过service获取不到metric /root/envoy/envoy.yaml
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 0.0.0.0 # 这里一定要改成0.0.0.0，而不能是127.0.0.1
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite: sealyun.com
cluster: service_google
http_filters:
- name: envoy.router
clusters:
- name: service_sealyun
connect_timeout: 0.25s
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
hosts:
- socket_address:
address: sealyun.com
port_value: 443
tls_context: { sni: sealyun.com }使用ServiceMonitor
envoyServiceMonitor.yaml:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: envoy
name: envoy
namespace: monitoring # 这个可以与service不在一个namespace中
spec:
endpoints:
- interval: 15s
port: metrics # envoy service的端口名
path: /stats/prometheus # 数据源path
namespaceSelector:
matchNames: # envoy service所在namespace
- default
selector:
matchLabels:
app: envoy # 选择envoy servicecreate成功后我们就可以看到envoy的数据源了：
然后就可以看到metric了：
然后就可以在grafana上进行一些配置了，promethues相关使用不是本文讨论的对象
告警配置
alert manager配置
[root@dev-86-201 envoy]# kubectl get secret -n monitoring
NAME TYPE DATA AGE
alertmanager-main Opaque 1 27d我们可以看到这个secrect，看下里面具体内容：
[root@dev-86-201 envoy]# kubectl get secret alertmanager-main -o yaml -n monitoring
apiVersion: v1
data:
alertmanager.yaml: Imdsb2JhbCI6IAogICJyZXNvbHZlX3RpbWVvdXQiOiAiNW0iCiJyZWNlaXZlcnMiOiAKLSAibmFtZSI6ICJudWxsIgoicm91dGUiOiAKICAiZ3JvdXBfYnkiOiAKICAtICJqb2IiCiAgImdyb3VwX2ludGVydmFsIjogIjVtIgogICJncm91cF93YWl0IjogIjMwcyIKICAicmVjZWl2ZXIiOiAibnVsbCIKICAicmVwZWF0X2ludGVydmFsIjogIjEyaCIKICAicm91dGVzIjogCiAgLSAibWF0Y2giOiAKICAgICAgImFsZXJ0bmFtZSI6ICJEZWFkTWFuc1N3aXRjaCIKICAgICJyZWNlaXZlciI6ICJudWxsIg==
kind: Secretbase64解码一下：
"global":
"resolve_timeout": "5m"
"receivers":
- "name": "null"
"route":
"group_by":
- "job"
"group_interval": "5m"
"group_wait": "30s"
"receiver": "null"
"repeat_interval": "12h"
"routes":
- "match":
"alertname": "DeadMansSwitch"
"receiver": "null"所以配置alertmanager就非常简单了，就是创建一个secrect即可 如alertmanager.yaml:
global:
smtp_smarthost: 'smtp.qq.com:465'
smtp_from: '4174785153@qq.com'
smtp_auth_username: '12785153@qq.com'
smtp_auth_password: 'xxx' # 这个密码是开启smtp授权后生成的,下文有说怎么配置
smtp_require_tls: false
route:
group_by: ['alertmanager','cluster','service']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: 'fanux'
routes:
- receiver: 'fanux'
receivers:
- name: 'fanux'
email_configs:
- to: '4741785153@qq.com'
send_resolved: true
delete掉老的secret，根据自己的配置重新生成secret即可
kubectl delete secret alertmanager-main -n monitoring
kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring邮箱配置，以QQ邮箱为例
开启smtp pop3服务
照着操作即可，后面会弹框一个授权码，配置到上面的配置文件中 然后就可以收到告警了：
告警规则配置
prometheus operator自定义PrometheusRule crd去描述告警规则
[root@dev-86-202 shell]# kubectl get PrometheusRule -n monitoring
NAME AGE
prometheus-k8s-rules 6m直接edit这个rule即可，也可以再自己去创建个PrometheusRule
kubectl edit PrometheusRule prometheus-k8s-rules -n monitoring如我们在group里加一个告警：
spec:
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
- name: k8s.rules
rules:重启prometheuspod:
kubectl delete pod prometheus-k8s-0 prometheus-k8s-1 -n monitoring
然后在界面上就可以看到新加的规则：
举报
腾讯云开发者
