使用k8s-prometheus-adapter实现HPA

环境：

kubernetes 1.11+/openshift3.11

自定义metric HPA原理：

首选需要注册一个apiservice(custom metrics API)。

当HPA请求metrics时，kube-aggregator(apiservice的controller)会将请求转发到adapter，adapter作为kubernentes集群的pod，实现了Kubernetes resource metrics API and custom metrics API，它会根据配置的rules从Prometheus抓取并处理metrics，在处理(如重命名metrics等)完后将metric通过custom metrics API返回给HPA。最后HPA通过获取的metrics的value对Deployment/ReplicaSet进行扩缩容。

adapter作为extension-apiserver(即自己实现的pod)，充当了代理kube-apiserver请求Prometheus的功能。

如下是k8s-prometheus-adapter apiservice的定义，kube-aggregator通过下面的service将请求转发给adapter。v1beta1.custom.metrics.k8s.io是写在k8s-prometheus-adapter代码中的，因此不能任意改变。

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

部署：

• github下载k8s-prometheus-adapter
• 参照官方文档部署adapter：
  • pull镜像：directxman12/k8s-prometheus-adapter:latest，修改镜像tag并push到本地镜像仓库
  • 生成证书：运行如下shell脚本(来自官方)生成cm-adapter-serving-certs.yaml，并将其拷贝到manifests/目录下，该证书用于kube-aggregator与adapter通信时认证adapter。注意下面证书有效时间为5年(43800h)以及授权的域名。 #!/usr/bin/env bash # exit immediately when a command fails set -e # only exit with zero if all commands of the pipeline exit successfully set -o pipefail # error on unset variables set -u # Detect if we are on mac or should use GNU base64 options case $(uname) in Darwin) b64_opts='-b=0' ;; *) b64_opts='--wrap=0' esac go get -v -u github.com/cloudflare/cfssl/cmd/... export PURPOSE=metrics echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "ca-config.json" export SERVICE_NAME=custom-metrics-apiserver export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"' echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \ cfssl gencert -ca=ca.crt -ca-key=ca.key -config=ca-config.json - | cfssljson -bare apiserver cat <<-EOF > cm-adapter-serving-certs.yaml apiVersion: v1 kind: Secret metadata: name: cm-adapter-serving-certs data: serving.crt: $(base64 ${b64_opts} < apiserver.pem) serving.key: $(base64 ${b64_opts} < apiserver-key.pem) EOF 可以在custom-metrics-apiservice.yaml中设置insecureSkipTLSVerify: true时，kube-aggregator不会校验adapter的如上证书。如果需要启用校验，则需要在caBundle中添加openshift集群的ca证书(非openshift集群的自签证书会被认为是不可信任的证书)，将openshift集群master节点的/etc/origin/master/ca.crt进行base64转码黏贴到caBundle字段即可。 base64 ca.crt 也可以黏贴openshift集群master节点的/root/.kube/config文件中的clusters.cluster.certificate-authority-data字段
    • 创建命名空间：kubectl create namespace custom-metrics
  • openshift的kube-system下面可能没有role extension-apiserver-authentication-reader,如果不存在，则需要创建 apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: extension-apiserver-authentication-reader namespace: kube-system rules: - apiGroups: - "" resourceNames: - extension-apiserver-authentication resources: - configmaps verbs: - get
  • 修改custom-metrics-apiserver-deployment.yaml的--prometheus-url字段，指向正确的prometheus
  • 创建其他组件：kubectl create -f manifests/ 在部署时会创建一个名为custom-metrics-resource-reader的clusterRole，用于授权adapter读取kubernetes cluster的资源，可以看到其允许读取的资源为namespaces/pods/services apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: custom-metrics-resource-reader rules: - apiGroups: - "" resources: - namespaces - pods - services verbs: - get - list
• 部署demo：
  • 部署官方demo # cat sample-app.deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: sample-app labels: app: sample-app spec: replicas: 1 selector: matchLabels: app: sample-app template: metadata: labels: app: sample-app spec: containers: - image: docker-local.art.aliocp.csvw.com/openshift3/autoscale-demo:v0.1.2 name: metrics-provider ports: - name: http containerPort: 8080
  • 创建service apiVersion: v1 kind: Service metadata: labels: app: sample-app name: sample-app namespace: custom-metrics spec: ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: app: sample-app type: ClusterIP 在custom-metrics命名空间下验证可以获取到metrics curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics
• 部署serviceMonitor 由于HPA需要用到namespace和pod等kubernetes的资源信息，因此需要使用servicemonitor注册方式来为metrics添加这些信息
  • openshift Prometheus operator对servicemonitor的限制如下 serviceMonitorNamespaceSelector: matchExpressions: - key: openshift.io/cluster-monitoring operator: Exists serviceMonitorSelector: matchExpressions: - key: k8s-app operator: Exists
  • 因此需要给custom-metrics命名空间添加标签 oc label namespace custom-metrics openshift.io/cluster-monitoring=true
  • 在openshift-monitoring命名空间中创建service-monitor # cat service-monitor.yaml kind: ServiceMonitor apiVersion: monitoring.coreos.com/v1 metadata: name: sample-app labels: k8s-app: testsample app: sample-app spec: namespaceSelector: any: true selector: matchLabels: app: sample-app endpoints: - port: http
  • 添加权限 oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s oc adm policy add-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s -n custom-metrics
• 测试HPA
  • 创建HPA，表示1秒请求大于0.5个时开始扩容 # cat sample-app-hpa.yaml kind: HorizontalPodAutoscaler apiVersion: autoscaling/v2beta1 metadata: name: sample-app spec: scaleTargetRef: # point the HPA at the sample application # you created above apiVersion: apps/v1 kind: Deployment name: sample-app # autoscale between 1 and 10 replicas minReplicas: 1 maxReplicas: 10 metrics: # use a "Pods" metric, which takes the average of the # given metric across all pods controlled by the autoscaling target - type: Pods pods: # use the metric that you used above: pods/http_requests metricName: http_requests_per_second # target 500 milli-requests per second, # which is 1 request every two seconds targetAverageValue: 500m 通过oc describe hpa sample-app查看hpa是否运行正常
  • 持续执行命令curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics发出请求
  • 通过命令kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"查看其对应的value值，当其值大于500m时开始扩容 # oc get pod NAME READY STATUS RESTARTS AGE sample-app-6d55487cdd-dc6qz 1/1 Running 0 18h sample-app-6d55487cdd-w6bbb 1/1 Running 0 5m sample-app-6d55487cdd-zbdbr 1/1 Running 0 5m
  • 过段时间，当kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"的值持续低于500m时进行缩容，缩容时间由--horizontal-pod-autoscaler-downscale-stabilization指定，默认5分钟。 提供oc get hpa的TARGETS字段可以查看扩缩容比例 # oc get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE sample-app Deployment/sample-app 66m/500m 1 10 1 3h

Adapter config

部署adapter前需要配置adapter的rule，用于预处理metrics，默认配置为manifests/custom-metrics-config-map.yaml。adapter的配置主要分为4个:

• Discovery：指定需要处理的Prometheus的metrics。通过seriesQuery挑选需要处理的metrics集合，可以通过seriesFilters精确过滤metrics。 seriesQuery可以根据标签进行查找(如下)，也可以直接指定metric name查找 seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}' seriesFilters: - isNot: "^container_.*_seconds_total" seriesFilters： is: <regex>, 匹配包含该正则表达式的metrics. isNot: <regex>, 匹配不包含该正则表达式的metrics.
• Association：设置metric与kubernetes resources的映射关系，kubernetes resorces可以通过kubectl api-resources命令查看。overrides会将Prometheus metric label与一个kubernetes resource(下例为deployment)关联。需要注意的是该label必须是一个真实的kubernetes resource，如metric的pod_name可以映射为kubernetes的pod resource，但不能将container_image映射为kubernetes的pod resource，映射错误会导致无法通过custom metrics API获取正确的值。这也表示metric中必须存在一个真实的resource 名称，将其映射为kubernetes resource。 resources: overrides: microservice: {group: "apps", resource: "deployment"}
• Naming：用于将prometheus metrics名称转化为custom metrics API所使用的metrics名称，但不会改变其本身的metric名称，即通过curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics获得的仍然是老的metric名称。如果不需要可以不执行这一步。 # match turn any name <name>_total to <name>_per_second # e.g. http_requests_total becomes http_requests_per_second name: matches: "^(.*)_total$" as: "${1}_per_second" 如本例中HPA后续可以通过/apis/{APIService-name}/v1beta1/namespaces/{namespaces-name}/pods/*/http_requests_per_second获取metrics
• Querying：处理调用custom metrics API获取到的metrics的value，该值最终提供给HPA进行扩缩容 # convert cumulative cAdvisor metrics into rates calculated over 2 minutes metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)" metricsQuery 字段使用Go template将URL请求转变为Prometheus的请求，它会提取custom metrics API请求中的字段，并将其划分为metric name,group-resource,以及group-resource中的一个或多个objects，对应如下字段：
  • Series: metric名称
  • LabelMatchers: 以逗号分割的objects，当前表示特定group-resource加上命名空间的label(如果该group-resource 是namespaced的)
  • GroupBy：以逗号分割的label的集合，当前表示LabelMatchers中的group-resource label

  假设metrics http_requests_per_second如下 http_requests_per_second{pod="pod1",service="nginx1",namespace="somens"} http_requests_per_second{pod="pod2",service="nginx2",namespace="somens"} 当调用kubectl get --raw "/apis/{APIService-name}/v1beta1/namespaces/somens/pods/*/http_request_per_second"时，metricsQuery字段的模板的实际内容如下：

  • Series: "http_requests_total"
  • LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"
  • GroupBy:pod

  adapter使用字段rules和externalRules分别表示custom metrics和external metrics，如本例中 apiVersion: v1 kind: ConfigMap metadata: name: adapter-config namespace: openshift-monitoring data: config.yaml: | externalRules: - seriesQuery: '{namespace!="",pod!=""}' seriesFilters: [] resources: overrides: namespace: resource: namespace pod: resource: pod metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[22m])) by (<<.GroupBy>>) rules: - seriesQuery: '{namespace!="",pod!=""}' seriesFilters: [] resources: overrides: namespace: resource: namespace pod: resource: pod name: matches: "^(.*)_total" as: "${1}_per_second" metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

HPA的配置

HPA通常会根据type从aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io)的资源路径上拉取metrics

HPA支持的metrics类型有4种(下述为v2beta2的格式)：

• resource：目前仅支持cpu和memory。target可以指定数值(targetAverageValue)和比例(targetAverageUtilization)进行扩缩容 HPA从metrics.k8s.io获取resource metrics
• pods：custom metrics，这类metrics描述了pod类型，target仅支持按指定数值(targetAverageValue)进行扩缩容。targetAverageValue 用于计算所有相关pods上的metrics的平均值 type: Pods pods: metric: name: packets-per-second target: type: AverageValue averageValue: 1k HPA从custom.metrics.k8s.io获取custom metrics
• object：custom metrics，这类metrics描述了相同命名空间下的(非pod)类型。target支持通过value和AverageValue进行扩缩容，前者直接将metric与target比较进行扩缩容，后者通过metric/相关的pod数目与target比较进行扩缩容 type: Object object: metric: name: requests-per-second describedObject: apiVersion: extensions/v1beta1 kind: Ingress name: main-route target: type: Value value: 2k
• external：kubernetes 1.10+。这类metrics与kubernetes集群无关(pods和object需要与kubernetes中的某一类型关联)。与object类似，target支持通过value和AverageValue进行扩缩容。由于external会尝试匹配所有kubernetes资源的metrics，因此实际中不建议使用该类型。 HPA从external.metrics.k8s.io获取external metrics - type: External external: metric: name: queue_messages_ready selector: "queue=worker_tasks" target: type: AverageValue averageValue: 30
• 1.6版本支持多metrics的扩缩容，当其中一个metrics达到扩容标准时就会创建pod副本(当前副本<maxReplicas)

注：target的value的一个单位可以划分为1000份，每一份以m为单位，如500m表示1/2个单位。参见Quantity

kubernetes HPA的算法如下：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

当使用targetAverageValue 或targetAverageUtilization时，currentMetricValue会取HPA指定的所有pods的metric的平均值

Kubernetes metrics的获取

假设注册的APIService为custom.metrics.k8s.io/v1beta1，在注册好APIService后HorizontalPodAutoscaler controller会从以/apis/custom.metrics.k8s.io/v1beta1为根API的路径上抓取metrics。metrics的API path可以分为namespaced和non-namespaced类型的。通过如下方式校验HPA是否可以获取到metrics：

namespaced

• 获取指定namespace下指定object类型和名称的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}"

如获取monitor命名空间下名为grafana的pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/grafana/start_time_seconds"

• 获取指定namespace下所有特定object类型的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}"

如获取monitor命名空间下名为所有pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/*/start_time_seconds"

• 使用labelSelector可以选择带有特定label的object

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}?labelSelector={label-name}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}?labelSelector={label-name}"

non-namespaced

non-namespaced和namespaced的类似，主要有node，namespace，PersistentVolume等。non-namespaced访问有些与custom metrics API描述不一致。

• 访问object为namespace的方式如下如下

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/metrics/{metric-name...}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/{metric-name...}"

• 访问node的方式如下

 kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/nodes/{node-name}/{metric-name...}"

DEBUG:

• 使用如下方式查看注册的APIService发现的所有rules kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 如果获取失败，可以看下使用oc get apiservice v1beta1.custom.metrics.k8s.io -oyaml查看status和message的相关信息 如果获取到的resource为空，则需要校验deploy中的Prometheus url是否正确，是否有权限等
• 通过如下方式查看完整的请求过程(--v=8) kubectl get --raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}" --v=8
• 如果上述过程正确，但获取到的items为空
  • 首先保证k8s-prometheus-adapter的参数--metrics-relist-interval设置值大于Prometheus的参数scrape_interval
  • 确保k8s-prometheus-adapter rules的seriesQuery规则可以抓取到Prometheus的数据
  • 确保k8s-prometheus-adapter rules的metricsQuery规则可以抓取到计算出数据，此处需要注意的是，如果使用到了计算某段时间的数据，如果时间设置过短，可能导致没有数据生成

TIPS:

• 官方提供了End-to-end walkthrough，但需要采集的metrics中包含pod和namespace label，否则在官方默认配置下无法采集到metrics。
• Configuration Walkthroughs一步步讲解了如何配置adapter config
• 在goland里面使用如下参数可以远程调试adapter： --secure-port=6443 --tls-cert-file=D:\adapter\serving.crt --tls-private-key-file=D:\adapter\serving.key --logtostderr=true --prometheus-url=${prometheus-url} --metrics-relist-interval=70s --v=10 --config=D:\adapter\config.yaml --lister-kubeconfig=D:\adapter\k8s-config.yaml --authorization-kubeconfig=D:\adapter\k8s-config.yaml --authentication-kubeconfig=D:\adapter\k8s-config.yaml

参考：

Kubernetes pod autoscaler using custom metrics

Kubernetes API Aggregation Setup — Nuts & Bolts

Configure the Aggregation Layer

Aggregation

Setup an Extension API Server

OpenShift下的JVM监控