本文章主要参考walkthrough,aggregation和auth。涉及custom metric API的注册认证以及API server aggregation的相关知识。walkthrough中主要实现了Prometheus adapter的功能,Prometheus adapter主要从Prometheus以一定间隔收集可用的metrics,然后以特定的格式暴露该metrics。
强烈建议阅读官方文档:setup an extension API server
HorizontalPodAutoscaler控制器可以通过两种方式获取metrics:通过Heapster接入方式和REST client接入方式。其中REST client方式即获取custom metrics的方式(参见:Horizontal Pod Autoscaler)
walkthrough中的主要步骤如下:
API server的flag中启用如下内容:
kube-controller-manager的flag中启用如下内容:
--horizontal-pod-autoscaler-use-rest-clients=true
--horizontal-pod-autoscaler-sync-period=10s //default 30s
--master=<apiserver-address>:<port> //port should be 8080
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true` # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
注:根据官方文档setup an extension API server,需要给serviceaccount绑定ClusterRole system:auth-delegator,该ClusterRole是为了插件API server向main API server请求认证和授权检查,认证方式为Webhook Token Authentication。绑定方式参考custom-metrics.yaml
上述流程中主要涉及的难点有:Prometheus和adapter的原理,证书的生成,extension-apiserver-authentication-role和aggregation
prometheus获取用户metrics使用的是service-discovery机制,参见kubernetes_sd_config(The service
role discovers a target for each service port for each service. This is generally useful for blackbox monitoring of a service. The address will be set to the Kubernetes DNS name of the service and respective service port)
Prometheus adapter的原理类似heapster,请求收集各prometheus的数据,并通过REST发送给HPA controller。prometheus的数据可以由prometheus client产生
同时还有--proxy-client-cert-file 和--proxy-client-key-file指定的代理证书。更多描述参见Configure the aggregation layer (There are a few setup requirements for getting the aggregation layer working in your environment to support mutual TLS auth between the proxy and extension apiservers. Kubernetes and the kube-apiserver have multiple CAs, so make sure that the proxy is signed by the aggregation layer CA and not by something else, like the master CA),也可以在插件的API server flag中传入--authentication-skip-lookup,但这样会同时关闭client认证,可以通过手动传入--client-ca-file启用client认证
关于extension-apiserver-authentication-role的表述参见Serving Certificates, Authentication, and Authorization,该role用于插件API server在默认条件下获取kube-system命名空间中的extension-apiserver-authentication ConfigMap,extension-apiserver-authentication ConfigMap中包含了main api server使用--client-ca-file指定的client CA和--requestheader-client-ca-file指定的RequestHeader client CA 。这种方式下,相同的client证书既可以用于k8s的认证,也可以用于插件API server的认证。
用于注册custom API,作为代理使用。参见aggregation
总结:
产生custom metrics的app模板中的annotations字段定义了暴露给prometheus的metrics的端口和路径;prometheus的配置中的scrape_configs字段通过标签定义了需要抓取的内容,与app中的annotations字段相呼应;custom metric API注册模板中指定了API的组和版本,以及该API对应的prometheus和prometheus adapter的service; 这样整个流程可以描述为:app产生custom metrics;prometheus抓取app产生的custom metrics;prometheus adapter请求并缓存更新prometheus的metrics,后续上报给kubernetes。从配置中可以看到prometheus并没有配置认证,而prometheus adapter则配置了与kubernetes交互的认证信息
流程图如下,aggregator通过service名称连接到APIService中指定的服务获取metric
参考:
How to build a Kubernetes Horizontal Pod Autoscaler using custom metrics
Configure Kubernetes Autoscaling With Custom Metrics
Monitoring Kubernetes performance metrics
https://github.com/slok/prometheus-python/tree/master/examples
https://www.slideshare.net/brianbrazil/python-ireland-monitoring-your-python-with-prometheus
https://godoc.org/github.com/prometheus/client_golang/prometheus https://www.cnblogs.com/gaorong/p/7881203.html