前面系列文章中:
•Prometheus Operator 与 kube-prometheus 之一 - 简介 - 东风微鸣技术博客 (ewhisper.cn)[1]•监控 Kubernetes 集群证书过期时间的三种方案 - 东风微鸣技术博客 (ewhisper.cn)[2]
介绍了 Prometheus Operator 相比 原生 Prometheus 的一些优势, 其已经被各大厂商和流行开源云组件广泛采用. 推荐使用.
但是实战中, 可能并不是所有组件都在 K8S 集群内, 如: LB、DB、全局DNS、云服务...
如何用 Prometheus Operator 监控它们? 这里有以下几种方案(算不上方案, 小技巧而已)
如上文, 这里的 K8s 集群外服务, 指的是一些如 LB、DB、全局DNS、云服务... 的静态服务.
针对此类服务, 有以下监控方案:
1.通过 Prometheus Operator CR - prometheus
spec;1.这种方案和 Prometheus 其他配置耦合性较高;2.通过 external name Service
+ ServiceMonitor
1.这种方案有个前提, 即: 被监控的服务是域名;3.通过 Service
+ Endpoint
+ ServiceMonitor
1.这种方案的适应性较强, 耦合性也较低. 推荐. 👍️4.如果是 BlackboxProbe 类的监控, 即监控: Endpoint(HTTP/S、DNS、TCP、ICMP 和 grpc)的各种参数,包括 HTTP 响应时间、DNS 查询延迟、SSL 证书过期信息、TLS 版本等等。可以直接使用 Probe
CR, 前文: 如何使用 Blackbox Exporter 监控 URL? - 东风微鸣技术博客 (ewhisper.cn)[3] 已经提过了, 本次就不再赘述.
prometheus
spec简而言之, 就是直接在 prometheus
spec 中加入类似这样的静态配置(static_configs):
static_configs:
- targets:
- SERVICE-FQDN
具体配置示例如下:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: monitor-kube-prometheus-st-prometheus
spec:
additionalScrapeConfigs:
- job_name: external
metrics_path: /metrics
static_configs:
- targets:
- <IP>:<PORT>
Service
+ ServiceMonitor
利用 Kubernetes 的 Externalname Serivce[4], 将服务映射到 DNS 名称, 而不是典型的选择算符,例如 my-service 或者 cassandra。
配置 Externalname Service:
apiVersion: v1
kind: Service
metadata:
name: gpu-metrics-svc
namespace: monitoring
labels:
k8s-app: gpu-metrics
spec:
type: ExternalName
externalName: <gpu-machine-ip>
clusterIP: ''
ports:
- name: metrics
port: 9100
protocol: TCP
targetPort: 9100
配置指向该 Service 的 ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: gpu-metrics-sm
labels:
k8s-app: gpu-metrics
prometheus: kube-prometheus
spec:
selector:
matchLabels:
k8s-app: gpu-metrics
namespaceSelector:
matchNames:
- monitoring
endpoints:
- port: metrics
interval: 10s
honorLabels: true
Service
+ Endpoint
+ ServiceMonitor
通过 Service
+ Endpoint
方式, 明确将外部服务映射为内部 Service.
举例如下:
kind: Service
apiVersion: v1
metadata:
name: external-es-exporter
labels:
app: elasticsearch
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: metrics
port: 9114
protocol: TCP
targetPort: 9114
---
apiVersion: v1
kind: Endpoints
metadata:
name: external-log-es-exporter
labels:
app: elasticsearch
namespace: monitoring
subsets:
- addresses:
- ip: <elasticsearch_ip_1>
- ip: <elasticsearch_ip_2>
- ip: <elasticsearch_ip_3>
ports:
- name: metrics
port: 9114
protocol: TCP
类似方案二, 再创建对应的 ServiceMonitor 即可:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
app: elasticsearch
namespaceSelector:
matchNames:
- monitoring
endpoints:
- port: metrics
path: /metrics
interval: 30s
这样虽然绕了一些, 但是可以保证, 修改组件 A 的监控的时候, 完全不会影响到组件 B 的配置; 另外, 也不会影响到 Prometheus 其他的监控.
配置更精确; 粒度更细; 耦合度更低.
🎉🎉🎉
[1]
Prometheus Operator 与 kube-prometheus 之一 - 简介 - 东风微鸣技术博客 (ewhisper.cn): https://ewhisper.cn/posts/14778/
[2]
监控 Kubernetes 集群证书过期时间的三种方案 - 东风微鸣技术博客 (ewhisper.cn): https://ewhisper.cn/posts/44110/
[3]
如何使用 Blackbox Exporter 监控 URL? - 东风微鸣技术博客 (ewhisper.cn): https://ewhisper.cn/posts/26225/#Blackbox-%20%E7%9A%84%20-Prometheus-Scrape-%20%E9%85%8D%E7%BD%AE
[4]
Externalname Serivce: https://kubernetes.io/zh-cn/docs/concepts/services-networking/service/#externalname
[5]
Scrape external service with FQDN · Issue #3204 · prometheus-operator/prometheus-operator (github.com): https://github.com/prometheus-operator/prometheus-operator/issues/3204
[6]
kubernetes - How to monitor external service in prometheus-operator - Stack Overflow: https://stackoverflow.com/questions/56848428/how-to-monitor-external-service-in-prometheus-operator
[7]
Prometheus Operator — How to monitor an external service | by Ido Braunstain | DevOps College: https://devops.college/prometheus-operator-how-to-monitor-an-external-service-3cb6ac8d5acb
[8]
Monitor external services with the prometheus operator | jpweber blog: https://jpweber.io/blog/monitor-external-services-with-the-prometheus-operator/
[9]
prometheus operator scrape external target for HAProxy — xnum's blog: https://xnum.github.io/2021/07/prome-op-external-target/