使用 helm 安装 Prometheus Operator。使用 helm 安装后,会在 Kubernetes 集群中创建、配置和管理 Prometheus 集群,chart 中包含多种组件:
Prometheus Operator 架构图如下:
上面架构图中,各组件以不同的方式运行在 Kubernetes 集群中:
操作系统:centos 7.6
ip地址:192.168.31.148
k8s版本:v1.16.3
主机名:k8s-master
配置:2核4g
helm:v2.16.6
操作系统:centos 7.6
ip地址:192.168.31.179
k8s版本:v1.16.3
主机名:k8s-node01
配置:4核8g
Prometheus Operator 所有的组件都打包成 Helm Chart,安装部署非常方便。
helm install --name prometheus-operator --set rbacEnable=true --namespace=monitoring stable/prometheus-operator
--
查看状态
# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 98m
pod/prometheus-operator-grafana-857dfc5fc8-vdnff 2/2 Running 0 99m
pod/prometheus-operator-kube-state-metrics-66b4c95cd9-mz8nt 1/1 Running 0 99m
pod/prometheus-operator-operator-56964458-8sspk 2/2 Running 0 99m
pod/prometheus-operator-prometheus-node-exporter-dcf5p 1/1 Running 0 99m
pod/prometheus-operator-prometheus-node-exporter-nv6ph 1/1 Running 0 99m
pod/prometheus-prometheus-operator-prometheus-0 3/3 Running 1 98m
确保pod状态运行正常
将type类型值改为NodePort
kubectl edit svc prometheus-operator-grafana -n monitoring
kubectl edit svc prometheus-operator-alertmanager -n monitoring
kubectl edit svc prometheus-operator-prometheus -n monitoring
查看修改后的访问类型
# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 100m
prometheus-operated ClusterIP None <none> 9090/TCP 100m
prometheus-operator-alertmanager NodePort 10.1.238.78 <none> 9093:31765/TCP 102mprometheus-operator-grafana NodePort 10.1.125.228 <none> 80:30284/TCP 102mprometheus-operator-kube-state-metrics ClusterIP 10.1.187.129 <none> 8080/TCP 102m
prometheus-operator-operator ClusterIP 10.1.242.61 <none> 8080/TCP,443/TCP 102m
prometheus-operator-prometheus NodePort 10.1.156.181 <none> 9090:30268/TCP 102m
prometheus-operator-prometheus-node-exporter ClusterIP 10.1.226.134 <none> 9100/TCP 102m
prometheus 需要访问 kubelet 的 10255 端口获取 metrics。但是默认情况下 10255 端口是不开放的,会导致 prometheus 上有 unhealthy。打开只读端口需要编辑所有节点的vi /var/lib/kubelet/config.yaml (我这里有图形界面可以用gedit命令)文件,加入以下内容
port: 10250
readOnlyPort: 10255 #增加此行
重启 kubelet 服务
systemctl restart kubelet.service
查看grafana随机端口
# kubectl get svc -n monitoring|grep grafana
prometheus-operator-grafana NodePort 10.1.125.228 <none> 80:30284/TCP 106m
可以看到,它的随机端口为:30284
访问grafana,地址为:http://nodeip:30284/
默认的用户名/密码为:admin/prom-operator,登陆后,点击Manage,如下图:
效果如下:
安装Ingress
kubectl apply -f https://kuboard.cn/install-script/v1.17.x/nginx-ingress.yaml
新建文件prometheus-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: monitoring
name: prometheus-ingress
spec:
rules:
- host: grafana.domain.com
http:
paths:
- backend:
serviceName: prometheus-operator-grafana
servicePort: 80
- host: prometheus.domain.com
http:
paths:
- backend:
serviceName: prometheus-operator-prometheus
servicePort: 9090
- host: alertmanager.domain.com
http:
paths:
- backend:
serviceName: prometheus-operator-alertmanager
servicePort: 9093
执行命令
kubectl apply -f prometheus-ingress.yaml
修改hosts文件:
192.168.31.179 grafana.domain.com
192.168.31.179 prometheus.domain.com
192.168.31.179 alertmanager.domain.com
注意:192.168.31.179 是Node节点ip
访问以下几个url,是否正常。
http://grafana.domain.com
http://prometheus.domain.com
http://alertmanager.domain.com
效果如下:
Prometheus Operator某些图表没有数据,需要修改配置文件才行。
prometheus 通过 4001 端口访问 etcd metrics,但是 etcd 默认监听 2379。解决方法是在 /etc/kubernetes/manifests/etcd.yaml
增加k8s-app: etcd-server
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
k8s-app: etcd-server
component: etcd
tier: control-plane
重启kubelet
systemctl restart kubelet.service
注意:经验证,此方法无效。etcd依然没有图表数据。
由于 kube-controller-manager 和 kube-scheduler 默认监听 127.0.0.1 ,prometheus 无法通过本机地址获取数据,需要修改kube-controller-manager 和 kube-scheduler 监听地址。
解决办法如下:/etc/kubernetes/manifests/kube-controller-manager.yaml 增加 k8s-app: kube-controller-manager
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
k8s-app: kube-controller-manager
component: kube-controller-manager
tier: control-plane
重启kubelet
systemctl restart kubelet.service
访问图表
/etc/kubernetes/manifests/kube-scheduler.yaml增加 k8s-app: kube-scheduler
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
k8s-app: kube-scheduler
component: kube-scheduler
tier: control-plane
重启kubelet
systemctl restart kubelet.service
访问图表
本文参考链接: