K8s部署metric-server页面不显示，报错unable to fully scrape metrics

院长技术

发布于 2021-04-30 14:30:16

2.3K0

发布于 2021-04-30 14:30:16

日志报错内容

[root@k8s-master1 ~]#kubectl logs -n kube-system metrics-server-77f9d68d78-s6drm 
I0428 02:36:28.023963       1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0428 02:36:28.863903       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0428 02:36:28.863936       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0428 02:36:28.863963       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0428 02:36:28.863980       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0428 02:36:28.863916       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0428 02:36:28.864275       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0428 02:36:28.864738       1 secure_serving.go:197] Serving securely on [::]:4443
I0428 02:36:28.865116       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0428 02:36:28.865135       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I0428 02:36:28.964294       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0428 02:36:28.964331       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0428 02:36:28.964374       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
E0428 02:36:29.110079       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node k8s-node2: unable to fetch metrics from node k8s-node2: Get "https://42.51.80.225:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.225 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master1: unable to fetch metrics from node k8s-master1: Get "https://42.51.80.221:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.221 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master3: unable to fetch metrics from node k8s-master3: Get "https://42.51.80.223:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.223 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node1: unable to fetch metrics from node k8s-node1: Get "https://42.51.80.224:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.224 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node3: unable to fetch metrics from node k8s-node3: Get "https://42.51.80.226:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.226 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master2: unable to fetch metrics from node k8s-master2: Get "https://42.51.80.222:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.222 because it doesn't contain any IP SANs]
E0428 02:37:29.085847       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node k8s-node1: unable to fetch metrics from node k8s-node1: Get "https://42.51.80.224:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.224 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node2: unable to fetch metrics from node k8s-node2: Get "https://42.51.80.225:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.225 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node3: unable to fetch metrics from node k8s-node3: Get "https://42.51.80.226:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.226 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master1: unable to fetch metrics from node k8s-master1: Get "https://42.51.80.221:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.221 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master3: unable to fetch metrics from node k8s-master3: Get "https://42.51.80.223:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.223 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master2: unable to fetch metrics from node k8s-master2: Get "https://42.51.80.222:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 42.51.80.222 because it doesn't contain any IP SANs]

解决方法

在metrics-server-deployment.yaml中aegs部分添加
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls  #添加
        - --metric-resolution=30s #添加
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.2
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir

重新apply资源

[root@k8s-master1 ~]#kubectl apply -f metricserver.yaml 
serviceaccount/metrics-server unchanged
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged
service/metrics-server unchanged
deployment.apps/metrics-server configured
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged

查看Pod状态,发现Pod正在更新

[root@k8s-master1 ~]#kubectl get pods -A
NAMESPACE              NAME                                         READY   STATUS        RESTARTS   AGE
default                curl                                         1/1     Running       0          17h
default                nginx                                        1/1     Running       0          17h
kube-system            calico-kube-controllers-69496d8b75-lf6c9     1/1     Running       0          19h
kube-system            calico-node-2plkn                            1/1     Running       0          18h
kube-system            calico-node-9f872                            1/1     Running       0          18h
kube-system            calico-node-b4265                            1/1     Running       0          18h
kube-system            calico-node-ggkdp                            1/1     Running       0          19h
kube-system            calico-node-rqdjh                            1/1     Running       0          18h
kube-system            calico-node-vsd7w                            1/1     Running       0          18h
kube-system            coredns-6c76c8bb89-9cpsv                     1/1     Running       0          19h
kube-system            coredns-6c76c8bb89-pwwmd                     1/1     Running       0          19h
kube-system            etcd-k8s-master1                             1/1     Running       2          19h
kube-system            etcd-k8s-master2                             1/1     Running       2          18h
kube-system            etcd-k8s-master3                             1/1     Running       1          18h
kube-system            kube-apiserver-k8s-master1                   1/1     Running       4          19h
kube-system            kube-apiserver-k8s-master2                   1/1     Running       2          18h
kube-system            kube-apiserver-k8s-master3                   1/1     Running       1          18h
kube-system            kube-controller-manager-k8s-master1          1/1     Running       14         19h
kube-system            kube-controller-manager-k8s-master2          1/1     Running       14         18h
kube-system            kube-controller-manager-k8s-master3          1/1     Running       15         18h
kube-system            kube-proxy-75jbb                             1/1     Running       0          18h
kube-system            kube-proxy-cdq2f                             1/1     Running       0          18h
kube-system            kube-proxy-jp6wb                             1/1     Running       0          18h
kube-system            kube-proxy-pdbpn                             1/1     Running       0          19h
kube-system            kube-proxy-qkr2q                             1/1     Running       0          18h
kube-system            kube-proxy-sppj8                             1/1     Running       0          18h
kube-system            kube-scheduler-k8s-master1                   1/1     Running       12         19h
kube-system            kube-scheduler-k8s-master2                   1/1     Running       16         18h
kube-system            kube-scheduler-k8s-master3                   1/1     Running       13         18h
kube-system            metrics-server-77f9d68d78-s6drm              0/1     Terminating   0          4m29s
kube-system            metrics-server-84d67bdb-b5qzk                1/1     Running       0          19s
kubernetes-dashboard   dashboard-metrics-scraper-79c5968bdc-h8wkg   1/1     Running       0          17h
kubernetes-dashboard   kubernetes-dashboard-9f9799597-9kkbw         1/1     Running       4          17h