前往小程序，Get更优阅读体验！
立即前往
文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
首页
学习
活动
专区
工具
TVP最新优惠活动
返回腾讯云官网
社区首页 >专栏 >《Prometheus监控实战》第12章 监控Kubernetes

《Prometheus监控实战》第12章 监控Kubernetes

作者头像
yeedomliu
发布于 2019-12-19 17:27:12
2K0
发布于 2019-12-19 17:27:12
举报
文章被收录于专栏：yeedomliuyeedomliu

第12章 监控Kubernetes

  • 我们会把Tornado、Prometheus部署到Kubernetes集群中（https://github.com/turnbullpress/tornado-api）

12.1 Kubernetes集群

  • Kubernetes集群名为tornado.quicknuke.com，使用kops构建了该集群（https://github.com/kubernetes/kops），可以在Github上找到集群配置。它有3个主节点和6个工作节点，所有节点都在三个可用区之间分布
  • 代码清单：kops创建集群命令
代码语言：javascript
复制
kops create cluster \
  --node-count 6 \
  --zones us-east-2a,us-east-2b,us-east-2c \
  --master-zones us-east-2a,us-east-2b,us-east-2c \
  --node-size t2.micro \
  --topology private \
  --networking kopeio-vxlan \
  --api-loadbalancer-type=public \
  --bastion \
  tornado.quicknuke.com

12.2 在Kubernetes上运行Prometheus

  • 在Kubernetes上部署Prometheus有多种方法，其中哪一种方法最适合很大程度上取决于你的环境。比如，你可以构建自己的部署然后通过服务的形式暴露Prometheus，这种方法已经有一些配置可供参考（https://github.com/kayrus/prometheus-kubernetes），或者可以使用CoreOS的PrometheusOperator（https://github.com/coreos/prometheus-operator）
  • 我们选择了手动创建部署和服务的方法，如下所示
代码语言：javascript
复制
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
    component: core
  annotations:
    prometheus.io/scrape: 'true'
    external-dns.alpha.kubernetes.io/hostname: prometheus.quicknuke.com.
spec:
  type: LoadBalancer
  ports:
    - port: 9090
      targetPort: webui
      name: webui
  selector:
    app: prometheus
    component: core
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-core
  namespace: monitoring
  labels:
    app: prometheus
    component: core
spec:
  replicas: 1
  template:
    metadata:
      name: prometheus-main
      labels:
        app: prometheus
        component: core
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.1.0
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--web.enable-lifecycle"
          ports:
            - containerPort: 9090
              name: webui
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus
            - name: prometheus-rules-volume
              mountPath: /var/prometheus/rules
            - name: prometheus-storage-volume
              mountPath: /prometheus/
        - name: watch
          image: weaveworks/watch:master-5b2a6e5
          imagePullPolicy: IfNotPresent
          args: ["-v", "-t", "-p=/etc/prometheus", "-p=/var/prometheus", "curl", "-X", "POST", "--fail", "-o", "-", "-sS", "http://localhost:9090/-/reload"]
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus
            - name: prometheus-rules-volume
              mountPath: /var/prometheus/rules
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-rules-volume
          configMap:
            name: prometheus-rules-conf
        - name: prometheus-storage-volume
          emptyDir: {}
  • 我们使用ConfigMaps来配置Prometheus服务器和管理规则，并在部署中将它们作为卷挂载
代码语言：javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval:     15s
      evaluation_interval: 15s

    alerting:
      alertmanagers:
      - kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: alertmanager
          action: keep
        - source_labels: [__meta_kubernetes_namespace]
          regex: monitoring
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          action: keep
          regex: cluster

    rule_files:
      - "/var/prometheus/rules/*_rules.yml"
      - "/var/prometheus/rules/*_alerts.yml"

    scrape_configs:
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

    - job_name: 'kubernetes-apiservers'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
  • 我们在Kubernetes上部署了包含3个Alertmanager节点的集群，并将所有这些资源都放在名为monitoring的命名空间
代码语言：javascript
复制
apiVersion: v1
kind: Service
metadata:
  name: alertmanager-webui
  namespace: monitoring
  labels:
    app: alertmanager
    component: core
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9093"
    external-dns.alpha.kubernetes.io/hostname: alertmanager.quicknuke.com.
spec:
  type: LoadBalancer
  ports:
    - port: 9093
      name: metrics
  selector:
    app: alertmanager
    component: core
---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: alertmanager
    component: core
spec:
  ports:
  - port: 9093
    name: cluster
  type: ClusterIP
  clusterIP: None
  selector:
    app: alertmanager
    component: core
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: alertmanager
    component: core
spec:
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  selector:
    matchLabels:
      app: alertmanager
      component: core
  serviceName: alertmanager
  template:
    metadata:
      labels:
        app: alertmanager
        component: core
    spec:
      containers:
      - name: alertmanager
        image: quay.io/prometheus/alertmanager:master
        imagePullPolicy: IfNotPresent
        command:
        - "sh"
        - "-c"
        args:
        - /bin/alertmanager
            --config.file=/etc/alertmanager/config.yml
            --web.listen-address=0.0.0.0:9093
            --cluster.listen-address=0.0.0.0:8001
            --storage.path=/alertmanager
            --cluster.peer="alertmanager-0.alertmanager.monitoring.svc:8001"
            --cluster.peer="alertmanager-1.alertmanager.monitoring.svc:8001"
            --cluster.peer="alertmanager-2.alertmanager.monitoring.svc:8001"
            --log.level=debug
        ports:
        - containerPort: 9093
          name: web
          protocol: TCP
        - containerPort: 8001
          name: cluster
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /api/v1/status
            port: web
            scheme: HTTP
          failureThreshold: 10
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /api/v1/status
            port: web
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        volumeMounts:
        - name: alertmanager-config-volume
          mountPath: /etc/alertmanager/
        - name: alertmanager-data-volume
          mountPath: /alertmanager/
      volumes:
      - name: alertmanager-config-volume
        configMap:
          name: alertmanager-server-conf
      - name: alertmanager-data-volume
        emptyDir: {}

12.3 监控Kubernetes

  • 利用Prometheus为Kubernetes提供的内置服务发现机制（https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config）
代码语言：javascript
复制
# A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# `labelmap` relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.

# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses `endpoints` role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port `https`. This works for single API server deployments as
# well as HA API server deployments.
scrape_configs:
- job_name: 'kubernetes-apiservers'

  kubernetes_sd_configs:
  - role: endpoints

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https

  # This TLS & bearer token file config is used to connect to the actual scrape
  # endpoints for cluster components. This is separate to discovery auth
  # configuration because discovery & scraping are two separate concerns in
  # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  # the cluster. Otherwise, more config options have to be provided within the
  # <kubernetes_sd_config>.
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    # If your node certificates are self-signed or use a different CA to the
    # master CA, then disable certificate verification below. Note that
    # certificate verification is an integral part of a secure infrastructure
    # so this should only be disabled in a controlled environment. You can
    # disable certificate verification by uncommenting the line below.
    #
    # insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  # Keep only the default/kubernetes service endpoints for the https port. This
  # will add targets for each API server which Kubernetes adds an endpoint to
  # the default/kubernetes service.
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

# Scrape config for nodes (kubelet).
#
# Rather than connecting directly to the node, the scrape is proxied though the
# Kubernetes apiserver.  This means it will work if Prometheus is running out of
# cluster, or can't connect to nodes for some other reason (e.g. because of
# firewalling).
- job_name: 'kubernetes-nodes'

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https

  # This TLS & bearer token file config is used to connect to the actual scrape
  # endpoints for cluster components. This is separate to discovery auth
  # configuration because discovery & scraping are two separate concerns in
  # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  # the cluster. Otherwise, more config options have to be provided within the
  # <kubernetes_sd_config>.
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  kubernetes_sd_configs:
  - role: node

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics

# Scrape config for Kubelet cAdvisor.
#
# This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
# (those whose names begin with 'container_') have been removed from the
# Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
# retrieve those metrics.
#
# In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
# HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
# in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
# the --cadvisor-port=0 Kubelet flag).
#
# This job is not necessary and should be removed in Kubernetes 1.6 and
# earlier versions, or it will cause the metrics to be scraped twice.
- job_name: 'kubernetes-cadvisor'

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https

  # This TLS & bearer token file config is used to connect to the actual scrape
  # endpoints for cluster components. This is separate to discovery auth
  # configuration because discovery & scraping are two separate concerns in
  # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  # the cluster. Otherwise, more config options have to be provided within the
  # <kubernetes_sd_config>.
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  kubernetes_sd_configs:
  - role: node

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

# Example scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# for all or only some endpoints.
- job_name: 'kubernetes-service-endpoints'

  kubernetes_sd_configs:
  - role: endpoints

  relabel_configs:
  # Example relabel to scrape only endpoints that have
  # "example.io/should_be_scraped = true" annotation.
  #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
  #    action: keep
  #    regex: true
  #
  # Example relabel to customize metric path based on endpoints
  # "example.io/metric_path = <metric path>" annotation.
  #  - source_labels: [__meta_kubernetes_service_annotation_example_io_metric_path]
  #    action: replace
  #    target_label: __metrics_path__
  #    regex: (.+)
  #
  # Example relabel to scrape only single, desired port for the service based
  # on endpoints "example.io/scrape_port = <port>" annotation.
  #  - source_labels: [__address__, __meta_kubernetes_service_annotation_example_io_scrape_port]
  #    action: replace
  #    regex: ([^:]+)(?::\d+)?;(\d+)
  #    replacement: $1:$2
  #    target_label: __address__
  #
  # Example relabel to configure scrape scheme for all service scrape targets
  # based on endpoints "example.io/scrape_scheme = <scheme>" annotation.
  #  - source_labels: [__meta_kubernetes_service_annotation_example_io_scrape_scheme]
  #    action: replace
  #    target_label: __scheme__
  #    regex: (https?)
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

# Example scrape config for probing services via the Blackbox Exporter.
#
# The relabeling allows the actual service scrape endpoint to be configured
# for all or only some services.
- job_name: 'kubernetes-services'

  metrics_path: /probe
  params:
    module: [http_2xx]

  kubernetes_sd_configs:
  - role: service

  relabel_configs:
  # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
  #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
  #    action: keep
  #    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox-exporter.example.com:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name

# Example scrape config for probing ingresses via the Blackbox Exporter.
#
# The relabeling allows the actual ingress scrape endpoint to be configured
# for all or only some services.
- job_name: 'kubernetes-ingresses'

  metrics_path: /probe
  params:
    module: [http_2xx]

  kubernetes_sd_configs:
  - role: ingress

  relabel_configs:
  # Example relabel to probe only some ingresses that have "example.io/should_be_probed = true" annotation
  #  - source_labels: [__meta_kubernetes_ingress_annotation_example_io_should_be_probed]
  #    action: keep
  #    regex: true
  - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
    regex: (.+);(.+);(.+)
    replacement: ${1}://${2}${3}
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox-exporter.example.com:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_ingress_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_ingress_name]
    target_label: kubernetes_name

# Example scrape config for pods
#
# The relabeling allows the actual pod scrape to be configured
# for all the declared ports (or port-free target if none is declared)
# or only some ports.
- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  # Example relabel to scrape only pods that have
  # "example.io/should_be_scraped = true" annotation.
  #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
  #    action: keep
  #    regex: true
  #
  # Example relabel to customize metric path based on pod
  # "example.io/metric_path = <metric path>" annotation.
  #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
  #    action: replace
  #    target_label: __metrics_path__
  #    regex: (.+)
  #
  # Example relabel to scrape only single, desired port for the pod
  # based on pod "example.io/scrape_port = <port>" annotation.
  # Note that __address__ is modified here, so if pod containers' ports
  # are declared, they all will be ignored.
  #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
  #    action: replace
  #    regex: ([^:]+)(?::\d+)?;(\d+)
  #    replacement: $1:$2
  #    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

12.4 监控Kubernetes节点

  • 我们可以利用Kubernetes DaemonSet控制器在集群中的每个节点上自动部署pod。当你无法控制实例时，这个方法很有用——例如，如果你使用托管的Kubernetes解决方案（https://kubernetes.io/zh/docs/concepts/workloads/controllers/daemonset/）
  • 警告：使用这种方法需要特别注意！Node Exporter可访问许多root级别的资源，在Docker容器中运行它需要将这些资源挂载到容器中，对于systemd收集器来说，需要以root身份运行容器。这带来了潜在的安全风险，如果你无法接受该风险，则应将Node Exporter直接安装到实例上

12.4.1 Node Exporter DaemonSet

  • DaemonSet使用toleration（容忍）确保pod在所有节点上运行，可能也包含主节点。它非常适合监控或日志代理等项目
  • 注意：可以在Github上找到Node Exporter的完整配置（https://github.com/turnbullpress/prometheusbook-code/blob/master/12-13/node-exporter.yml）
代码语言：javascript
复制
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: scrape
    port: 9100
    protocol: TCP
  selector:
    app: node-exporter
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  template:
    metadata:
      labels:
        app: node-exporter
      name: node-exporter
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      hostNetwork: true
      hostPID: true
      hostIPC: true
      securityContext:
        runAsUser: 0
      containers:
      - image: prom/node-exporter:latest
        name: node-exporter
        volumeMounts:
          - mountPath: /run/systemd/private
            name: systemd-socket
            readOnly: true
        args:
          - "--collector.systemd"
          - "--collector.systemd.unit-whitelist=(docker|ssh|rsyslog|kubelet).service"
        ports:
          - containerPort: 9100
            hostPort: 9100
            name: scrape
        livenessProbe:
          httpGet:
            path: /metrics
            port: 9100
          initialDelaySeconds: 30
          timeoutSeconds: 10
          periodSeconds: 1
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /metrics
            port: 9100
          initialDelaySeconds: 10
          timeoutSeconds: 10
          periodSeconds: 2
      volumes:
        - hostPath:
            path: /run/systemd/private
          name: systemd-socket
  • 代码清单：Node Exporter DaemonSet toleration
代码语言：javascript
复制
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  template:
    metadata:
      labels:
        app: node-exporter
      name: node-exporter
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      hostNetwork: true
      hostPID: true
      hostIPC: true
      securityContext:
        runAsUser: 0
……………………
  • 使用toleration来确保pod也会被调度到Kubernetes主节点，而不仅是工作节点
  • 我们以用户0或root运行pod（这允许访问systemd），并且还启用了hostNetwork、hostPID和hostIPC，以指定实例的网络、进程和IPC命名空间在容器中可用。这些都是潜在的安全风险，你必须考虑是否可以承担此风险。如果这种风险是不可接受的，那么将Node Exporter放到实例的镜像中可能是一种更好的方法
  • 代码清单：Node Exporter DaemonSet容器
代码语言：javascript
复制
………………
      containers:
      - image: prom/node-exporter:latest
        name: node-exporter
        volumeMounts:
          - mountPath: /run/systemd/private
            name: systemd-socket
            readOnly: true
        args:
          - "--collector.systemd"
          - "--collector.systemd.unit-whitelist=(docker|ssh|rsyslog|kubelet).service"
        ports:
          - containerPort: 9100
            hostPort: 9100
            name: scrape
……………………
  • 使用DockerHub中的Node Exporter镜像prom/node_exporter（https://hub.docker.com/r/prom/node-exporter），并获取最新版本。我们还为实例上的目录`/run/systemd/private`挂载了一个卷，这允许Node Exporter访问systemd并在实例上收集systemd管理服务的服务状态
  • 为了帮助Node Exporter pod保持健康并提高它们的正常运行时间，我们还在Node Exporter容器中添加了liveness和readiness探针（https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/），其中liveness探针检测容器内应用程序的状态
代码语言：javascript
复制
……………………
        livenessProbe:
          httpGet:
            path: /metrics
            port: 9100
          initialDelaySeconds: 30
          timeoutSeconds: 10
          periodSeconds: 1
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /metrics
            port: 9100
          initialDelaySeconds: 10
          timeoutSeconds: 10
          periodSeconds: 2
………………
  • readiness探针可确认应用程序正常运行。这意味着，在将容器标记为可用并发送流量之前，HTTP GET可以连接到端口9100的/metrics路径。其余设置控制探针的行为，在检查准备就绪之前，它将等待10秒（initialDelaySecond），然后，它会每隔2秒（periodSeconds）检查一次。如果探针在10秒（timeoutSeconds）后超时超过5次（failureThreshold），则容器将被标记为Unready

12.4.2 Node Exporter服务

  • 创建一个服务来暴露Node Exporter，以便进行指标抓取
  • 代码清单：Node Exporter服务
代码语言：javascript
复制
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: scrape
    port: 9100
    protocol: TCP
  selector:
    app: node-exporter
  type: ClusterIP
  • 我们添加了一个注解prometheus.io/scrape:'true'（https://kubernetes.io/zh/docs/concepts/overview/working-with-objects/annotations/），作为服务的元数据，它将告诉Prometheus应该抓取这个服务

12.4.3 部署Node Exporter

  • 部署Node Exporter DaemonSet和服务
代码语言：javascript
复制
kubectl create -f ./node-exporter.yml -n monitoring
daemonset "node-exporter" created
service "node-exporter" created
  • 代码清单：默认命名空间
代码语言：javascript
复制
kubectl config set-context $(kubectl config current-context) --namespace=monitoring
  • 代码清单：检查Node Exporter pod是否运行
  • 集群中每个实例1个：3个主节点和6个工作节点。我们还可以看到Prometheus服务和Alertmanager的pod，分别是prometheus-core和alertmanager
  • 代码清单：捕捉日志检查Node Exporter pod日志确定是否正常运行
  • 代码清单：检查Node Exporter服务

12.4.4 Node Exporter作业

  • 我们还会控制Prometheus仅抓取具有注定注解prometheus.io/scrape（设置为true）的端点。然后，我们使用内置的Kubernetes服务发现来查找端点，并将它们作为Prometheus的潜在目标返回
  • 注意：所有这些工作都是基于Prometheus自带的Kubernetes作业案例
代码语言：javascript
复制
# A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# `labelmap` relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.

# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses `endpoints` role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port `https`. This works for single API server deployments as
# well as HA API server deployments.
scrape_configs:
  - job_name: 'kubernetes-apiservers'

    kubernetes_sd_configs:
      - role: endpoints

    # Default to scraping over https. If required, just disable this or change to
    # `http`.
    scheme: https

    # This TLS & bearer token file config is used to connect to the actual scrape
    # endpoints for cluster components. This is separate to discovery auth
    # configuration because discovery & scraping are two separate concerns in
    # Prometheus. The discovery auth config is automatic if Prometheus runs inside
    # the cluster. Otherwise, more config options have to be provided within the
    # <kubernetes_sd_config>.
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      # If your node certificates are self-signed or use a different CA to the
      # master CA, then disable certificate verification below. Note that
      # certificate verification is an integral part of a secure infrastructure
      # so this should only be disabled in a controlled environment. You can
      # disable certificate verification by uncommenting the line below.
      #
      # insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    # Keep only the default/kubernetes service endpoints for the https port. This
    # will add targets for each API server which Kubernetes adds an endpoint to
    # the default/kubernetes service.
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

  # Scrape config for nodes (kubelet).
  #
  # Rather than connecting directly to the node, the scrape is proxied though the
  # Kubernetes apiserver.  This means it will work if Prometheus is running out of
  # cluster, or can't connect to nodes for some other reason (e.g. because of
  # firewalling).
  - job_name: 'kubernetes-nodes'

    # Default to scraping over https. If required, just disable this or change to
    # `http`.
    scheme: https

    # This TLS & bearer token file config is used to connect to the actual scrape
    # endpoints for cluster components. This is separate to discovery auth
    # configuration because discovery & scraping are two separate concerns in
    # Prometheus. The discovery auth config is automatic if Prometheus runs inside
    # the cluster. Otherwise, more config options have to be provided within the
    # <kubernetes_sd_config>.
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    kubernetes_sd_configs:
      - role: node

    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

  # Scrape config for Kubelet cAdvisor.
  #
  # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
  # (those whose names begin with 'container_') have been removed from the
  # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
  # retrieve those metrics.
  #
  # In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
  # HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
  # in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
  # the --cadvisor-port=0 Kubelet flag).
  #
  # This job is not necessary and should be removed in Kubernetes 1.6 and
  # earlier versions, or it will cause the metrics to be scraped twice.
  - job_name: 'kubernetes-cadvisor'

    # Default to scraping over https. If required, just disable this or change to
    # `http`.
    scheme: https

    # This TLS & bearer token file config is used to connect to the actual scrape
    # endpoints for cluster components. This is separate to discovery auth
    # configuration because discovery & scraping are two separate concerns in
    # Prometheus. The discovery auth config is automatic if Prometheus runs inside
    # the cluster. Otherwise, more config options have to be provided within the
    # <kubernetes_sd_config>.
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    kubernetes_sd_configs:
      - role: node

    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

  # Example scrape config for service endpoints.
  #
  # The relabeling allows the actual service scrape endpoint to be configured
  # for all or only some endpoints.
  - job_name: 'kubernetes-service-endpoints'

    kubernetes_sd_configs:
      - role: endpoints

    relabel_configs:
      # Example relabel to scrape only endpoints that have
      # "example.io/should_be_scraped = true" annotation.
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
      #    action: keep
      #    regex: true
      #
      # Example relabel to customize metric path based on endpoints
      # "example.io/metric_path = <metric path>" annotation.
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_metric_path]
      #    action: replace
      #    target_label: __metrics_path__
      #    regex: (.+)
      #
      # Example relabel to scrape only single, desired port for the service based
      # on endpoints "example.io/scrape_port = <port>" annotation.
      #  - source_labels: [__address__, __meta_kubernetes_service_annotation_example_io_scrape_port]
      #    action: replace
      #    regex: ([^:]+)(?::\d+)?;(\d+)
      #    replacement: $1:$2
      #    target_label: __address__
      #
      # Example relabel to configure scrape scheme for all service scrape targets
      # based on endpoints "example.io/scrape_scheme = <scheme>" annotation.
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_scrape_scheme]
      #    action: replace
      #    target_label: __scheme__
      #    regex: (https?)
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

  # Example scrape config for probing services via the Blackbox Exporter.
  #
  # The relabeling allows the actual service scrape endpoint to be configured
  # for all or only some services.
  - job_name: 'kubernetes-services'

    metrics_path: /probe
    params:
      module: [http_2xx]

    kubernetes_sd_configs:
      - role: service

    relabel_configs:
      # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name

  # Example scrape config for probing ingresses via the Blackbox Exporter.
  #
  # The relabeling allows the actual ingress scrape endpoint to be configured
  # for all or only some services.
  - job_name: 'kubernetes-ingresses'

    metrics_path: /probe
    params:
      module: [http_2xx]

    kubernetes_sd_configs:
      - role: ingress

    relabel_configs:
      # Example relabel to probe only some ingresses that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_ingress_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_name

  # Example scrape config for pods
  #
  # The relabeling allows the actual pod scrape to be configured
  # for all the declared ports (or port-free target if none is declared)
  # or only some ports.
  - job_name: 'kubernetes-pods'

    kubernetes_sd_configs:
      - role: pod

    relabel_configs:
      # Example relabel to scrape only pods that have
      # "example.io/should_be_scraped = true" annotation.
      #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
      #    action: keep
      #    regex: true
      #
      # Example relabel to customize metric path based on pod
      # "example.io/metric_path = <metric path>" annotation.
      #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
      #    action: replace
      #    target_label: __metrics_path__
      #    regex: (.+)
      #
      # Example relabel to scrape only single, desired port for the pod
      # based on pod "example.io/scrape_port = <port>" annotation.
      # Note that __address__ is modified here, so if pod containers' ports
      # are declared, they all will be ignored.
      #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
      #    action: replace
      #    regex: ([^:]+)(?::\d+)?;(\d+)
      #    replacement: $1:$2
      #    target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
  • 代码清单：Kubernetes服务端点作业
代码语言：javascript
复制
- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
  • 我们称这个作业为kubernetes-service-endpoints，指定服务发现使用kubernetes_sd_discovery机制——这是内置的服务发现机制，专门用于监控Kubernetes。它向Kubernetes API查询符合特定搜索条件的目标
  • 由于Prometheus服务器在Kubernetes内部运行，因此我们能够以最少的配置自动获取与特定角色匹配的Kubernetes目标。节点、pod、服务和入口都有不同的角色，由参数role指定，我们要求服务发现返回所有Kubernetes端点。endpoints角色返回服务的所有已列出端点的目标，每个端点地址的每个端口都是一个目标
  • 代码清单：endpoints角色
代码语言：javascript
复制
endpoints
The endpoints role discovers targets from listed endpoints of a service. For each endpoint address one target is discovered per port. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

Available meta labels:

__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
__meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
__meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
__meta_kubernetes_endpoint_ready: Set to true or false for the endpoint's ready state.
__meta_kubernetes_endpoint_port_name: Name of the endpoint port.
__meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
__meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
__meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
If the endpoints belong to a service, all labels of the role: service discovery are attached.
For all targets backed by a pod, all labels of the role: pod discovery are attached.
  • 服务发现还会填充各种元数据
  • 第一条规则检查Node Exporter服务中设置的注解prometheus.io/scrape:'true'。在服务发现过程中prometheus.io/scrape注解会被转换为prometheus_io_scrape，以创建一个有效的标签名称，这是因为在Prometheus指标标签中点和斜杠不是合法字符。由于这是Kubernetes服务的注解，因此Prometheus服务进程还会在标签中添加__meta_kubernetes_service_annotation_前缀
  • 作业只保留具有元数据标签的目标，即__meta_kubernetes_service_annotation_（设置为true）。所有其他目标都会被丢弃，这使得你只抓取所需的端点
  • prometheus.io/scheme、prometheus.io/path和prometheus.io/port。如果这些标签存在，那么，它将使用这些注解的内容作为要抓取的scheme、path和port。这使我们能够从服务端点精确控制要抓取的内容，进而使作业变得更加灵活
  • 将__meta_kubernetes_service_label_app元数据标签映射为一个简单的app标签。下一个规则将__meta_kubernetes_namespace标签复制到kubernetes_namespace，将__meta_kubernetes_service_name元数据标签复制到kubernetes_name
  • 代码清单：将作业添加到用于Prometheus服务配置的ConfigMap
代码语言：javascript
复制
kubectl replace -f ./prom-config-map-v1.yml -n monitoring
  • 删除Prometheus pod，以便重新创建并加载新配置
  • 代码清单：监控服务
  • 这个作业非常有价值，因为我们只需要定义一次，未来所有Kubernetes服务端点都针被自动发现和监控

12.4.5 Node Exporter规则

  • 代码清单：Kubernetes可用性警报规则
代码语言：javascript
复制
- alert: KubernetesServiceDown
  expr: up{job="kubernetes-service-endpoints"} == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: Pod {{ $labels.instance }} is down!
- alert: KubernetesServicesGone
  expr:  absent(up{job="kubernetes-service-endpoints"})
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: No Kubernetes services are reporting!
    description: Werner Heisenberg says - OMG Where are my servicez?
  • 当kubernetes-service-endpoints作业的up指标值 为0时，将触发第一个警报，它表示Prometheus未能抓取任务。第二个警报检查服务是否消失，并使用absent函数检查up指标是否存在
  • 还使用node_systemd_unit_state指标为各个节点上监控的服务添加了警报规则，该指标跟踪systemd服务的状态
  • 代码清单：Kubernetes可用性警报规则
代码语言：javascript
复制
- alert: CriticalServiceDown
  expr: node_systemd_unit_state{state="active"} != 1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: Service {{ $labels.name }} failed to start.
    description: Service {{ $labels.instance }} failed to (re)start service {{ $labels.name }}.

12.5 Kubernetes

  • 有很多种方法可以监控Kubernetes本身，包括开源Kubernetes生态系统中的工具，如Heapster（https://github.com/kubernetes-retired/heapster）和Kube-state-metrics（https://github.com/kubernetes/kube-state-metrics）

12.5.1 Kube-state-metrics

  • 将在Kubernetes集群中使用部署和服务来安装Kube-state-metrics。部署使用Docker镜像Kube-state-metrics，并在集群的一个节点上运行。服务在端口8080上暴露指标，由于它是一项服务，因此我们可以利用在上一节中创建的Prometheus服务作业。当作业运行时，Prometheus将自动发现新的服务端点并开始抓取Kube-state-metrics
  • 代码清单：部署服务
代码语言：javascript
复制
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics
    protocol: TCP
  selector:
    app: kube-state-metrics
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  replicas: 1
  template:
    metadata:
      name: kube-state-metrics-main
      labels:
        app: kube-state-metrics
    spec:
      containers:
        - name: kube-state-metrics
          image: quay.io/coreos/kube-state-metrics:latest
          ports:
          - containerPort: 8080
            name: metrics
  • 代码清单：Prometheus服务作业
代码语言：javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval:     15s
      evaluation_interval: 15s

    alerting:
      alertmanagers:
      - kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: alertmanager
          action: keep
        - source_labels: [__meta_kubernetes_namespace]
          regex: monitoring
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          action: keep
          regex: cluster

    rule_files:
      - "/var/prometheus/rules/*_rules.yml"
      - "/var/prometheus/rules/*_alerts.yml"

    scrape_configs:
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

    - job_name: 'kubernetes-apiservers'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
  • 提示：你可以在文档 中查看Kube-state-metrics生成的指标的完整列表（https://prometheus.io/docs/prometheus/latest/querying/functions/）
  • 代码清单：Kube-state-metrics部署版本警报
代码语言：javascript
复制
- alert: TornadoRedisCacheMissesHigh
  expr: redis_keyspace_hits_total / (redis_keyspace_hits_total + redis_keyspace_misses_total) > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: Redis Server {{ $labels.instance }} Cache Misses are high.
  • 第一条规则检测部署是否成功。它将部署的运行版本与元数据中的版本进行比较。如果两者在5分钟内不相等，则会发出警报，指示部署失败
  • 代码清单：Kube-state-metrics部署副本未更新警报
代码语言：javascript
复制
- alert: DeploymentReplicasNotUpdated
  expr: ((kube_deployment_status_replicas_updated != kube_deployment_spec_replicas)
    or (kube_deployment_status_replicas_available != kube_deployment_spec_replicas))
    unless (kube_deployment_spec_paused == 1)
  for: 5m
  labels:
    severity: warning
  annotations:
    description: Replicas are not updated and available for deployment {{ $labels.namespace }}/{{ $labels.deployment }}
    summary: Deployment replicas are outdated
  • 假设部署没有暂停，确认更新或可用的副本是否与部署规范中的副本数相匹配
  • 代码清单：Kube-state-metrics 检查pod重启警报
代码语言：javascript
复制
- alert: PodzFrequentlyRestarting
  expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
  for: 10m
  labels:
    severity: warning
  annotations:
    description: Pod {{ $labels.namespace }}/{{ $labels.pod }} was restarted {{ $value }} times within the last hour
    summary: Pod is restarting frequently
  • 使用increase函数测量pod重启次数。increase函数测量范围向量中时间序列的增长率，这里是1小时。如果数值持续10分钟超过5，则会触发警报
increase

increase(v range-vector) calculates the increase in the time series in the range vector. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for. The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments. The following example expression returns the number of HTTP requests as measured over the last 5 minutes, per time series in the range vector:

代码语言：javascript
复制
increase(http_requests_total{job="api-server"}[5m])

increase should only be used with counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.

  • 我们还可以使用许多其他时间序列来监控Kubernetes。例如，可以使用kube_node_status_condition来确定Kubernetes节点的可用性。其它警报规则示例
  • 代码清单：警报规则示例kubernetes_alerts.yml
代码语言：javascript
复制
groups:
  - name: kubernetes_alerts
    rules:
      - alert: DeploymentGenerationOff
        expr: kube_deployment_status_observed_generation != kube_deployment_metadata_generation
        for: 5m
        labels:
          severity: warning
        annotations:
          description: Deployment generation does not match expected generation {{ $labels.namespace }}/{{ $labels.deployment }}
          summary: Deployment is outdated
      - alert: DeploymentReplicasNotUpdated
        expr: ((kube_deployment_status_replicas_updated != kube_deployment_spec_replicas)
          or (kube_deployment_status_replicas_available != kube_deployment_spec_replicas))
          unless (kube_deployment_spec_paused == 1)
        for: 5m
        labels:
          severity: warning
        annotations:
          description: Replicas are not updated and available for deployment {{ $labels.namespace }}/{{ $labels.deployment }}
          summary: Deployment replicas are outdated
      - alert: PodzFrequentlyRestarting
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          description: Pod {{ $labels.namespace }}/{{ $labels.pod }} was restarted {{ $value }} times within the last hour
          summary: Pod is restarting frequently
      - alert: KubeNodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 1h
        labels:
          severity: warning
        annotations:
          description: The Kubelet on {{ $labels.node }} has not checked in with the API,
            or has set itself to NotReady, for more than an hour
          summary: Node status is NotReady
      - alert: KubeManyNodezNotReady
        expr: count(kube_node_status_condition{condition="Ready",status="true"} == 0)
          > 1 and (count(kube_node_status_condition{condition="Ready",status="true"} ==
          0) / count(kube_node_status_condition{condition="Ready",status="true"})) > 0.2
        for: 1m
        labels:
          severity: critical
        annotations:
          description: '{{ $value }}% of Kubernetes nodes are not ready'
      - alert: APIHighLatency
        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"} > 4
        for: 10m
        labels:
          severity: critical
        annotations:
          description: the API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}
      - alert: APIServerErrorsHigh
        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m]) * 100 > 5
        for: 10m
        labels:
          severity: critical
        annotations:
          description: API server returns errors for {{ $value }}% of requests
      - alert: KubernetesAPIServerDown
        expr: up{job="kubernetes-apiservers"} == 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: Apiserver {{ $labels.instance }} is down!
      - alert: KubernetesAPIServersGone
        expr:  absent(up{job="kubernetes-apiservers"})
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: No Kubernetes apiservers are reporting!
          description: Werner Heisenberg says - OMG Where are my apiserverz?

12.5.2 Kube API

  • 我们还想创建一个作业来监控Kubernetes API本身。与API相关的指标将构成Kubernetes监控的核心，使我们能够监控集群的延迟、错误率和可用性
  • 代码清单：API服务器作业
代码语言：javascript
复制
- job_name: 'kubernetes-apiservers'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
  • 我们将作业命名为kubernetes-apiservers，使用https来抓取指标，并指定证书颁发机构和一个本地令牌文件，以对Kubernetes进行身份验证。我们再次使用Kubernetes发现，这次会返回Kubernetes endpoints列表。我们不会使用所有端点，并且重新标记配置使用keep操作，以在default命名空间中仅保留名为kubenetes的服务，即运行API的Kubernetes主节点
  • 代码清单：API服务器记录规则
代码语言：javascript
复制
- record: apiserver_latency_seconds:quantile
  expr: histogram_quantile(0.99, rate(apiserver_request_latencies_bucket[5m])) / 1e+06
  labels:
    quantile: "0.99"
  - record: apiserver_latency_seconds:quantile
    expr: histogram_quantile(0.9, rate(apiserver_request_latencies_bucket[5m])) / 1e+06
    labels:
      quantile: "0.9"
  - record: apiserver_latency_seconds:quantile
    expr: histogram_quantile(0.5, rate(apiserver_request_latencies_bucket[5m])) / 1e+06
    labels:
      quantile: "0.5"
  • 这里使用apiserver_request_latencies_bucket指标来计算延迟，该指标包含特定API资源、子资源和谓词的维度，用于衡量请求延迟。我们为第50、90和99百分位数创建了三个规则，将quantile设置为特定百分位数，然后使用histogram_quantile函数从指标桶中创建百分位数。我们已经指定了正在寻找的百分位数，例如0.99，然后计算了5分钟向量的速率rate，并将结果除以1e+06或1 000 000以获得微秒延迟

histogram_quantile(φ float, b instant-vector) calculates the φ-quantile (0 ≤ φ ≤ 1) from the buckets b of a histogram. (See histograms and summaries for a detailed explanation of φ-quantiles and the usage of the histogram metric type in general.) The samples in b are the counts of observations in each bucket. Each sample must have a label le where the label value denotes the inclusive upper bound of the bucket. (Samples without such a label are silently ignored.) The histogram metric type automatically provides time series with the _bucketsuffix and the appropriate labels. Use the rate() function to specify the time window for the quantile calculation. Example: A histogram metric is called http_request_duration_seconds. To calculate the 90th percentile of request durations over the last 10m, use the following expression:

代码语言：javascript
复制
histogram_quantile(0.9, rate(http_request_duration_seconds_bucket[10m]))

The quantile is calculated for each label combination in http_request_duration_seconds. To aggregate, use the sum() aggregator around the rate() function. Since the le label is required by histogram_quantile(), it has to be included in the by clause. The following expression aggregates the 90th percentile by job:

代码语言：javascript
复制
histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket[10m])) by (job, le))

To aggregate everything, specify only the le label:

代码语言：javascript
复制
histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket[10m])) by (le))

The histogram_quantile() function interpolates quantile values by assuming a linear distribution within a bucket. The highest bucket must have an upper bound of +Inf. (Otherwise, NaN is returned.) If a quantile is located in the highest bucket, the upper bound of the second highest bucket is returned. A lower limit of the lowest bucket is assumed to be 0 if the upper bound of that bucket is greater than 0. In that case, the usual linear interpolation is applied within that bucket. Otherwise, the upper bound of the lowest bucket is returned for quantiles located in the lowest bucket. If b contains fewer than two buckets, NaN is returned. For φ < 0, -Inf is returned. For φ > 1, +Inf is returned.

  • 代码清单：API高延迟警报
代码语言：javascript
复制
- alert: APIHighLatency
  expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"} > 4
  for: 10m
  labels:
    severity: critical
  annotations:
    description: the API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}
  • 警报使用了刚刚创建的apiserver_latency_seconds:quantile指标。我们使用标签来选择第99百分位数，不是log的子资源，以及任何不是WATCH、WATCHLIST、PROXY或CONNECT的谓词。如果任何指标的延迟持续10分钟超过4，则会触发警报
  • 代码清单：检测API高错误率警报
代码语言：javascript
复制
- alert: APIServerErrorsHigh
  expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m]) * 100 > 5
  for: 10m
  labels:
    severity: critical
  annotations:
    description: API server returns errors for {{ $value }}% of requests
  • 此警报计算API请求的错误率，使用正则表达式来匹配任何以5xx开头的错误。如果5分钟向量的百分比超过5%，则会触发警报
  • 代码清单：监控API服务器故障或不存在
代码语言：javascript
复制
- alert: KubernetesAPIServerDown
  expr: up{job="kubernetes-apiservers"} == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: Apiserver {{ $labels.instance }} is down!
  • 我们可以监控Kubernetes节点以及在节点上运行的Docker守护进程和容器

12.5.3 cAdvisor和节点

  • Kubernetes默认提供cAdvisor和特定节点的时间序列。我们可以创建一个作业来从每个节点的Kubernetes API中抓取这些时间序列。我们可以使用这些时间序列监控节点，以及每个节点上的Docker守护进程和容器
  • 代码清单：为cAdvisor添加一个作业
代码语言：javascript
复制
- job_name: 'kubernetes-cadvisor'
  scheme: https
  tls_config:
    insecure_skip_verify: true
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
    - role: node
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  • 这里将作业命名为kubernetes-cadvisor，并使用服务发现来返回node角色的Kubernetes节点列表。我们使用https来抓取指标，并指定证书颁发机构和一个本地令牌文件以对Kubernetes进行身份验证
  • 然后我们重新标记时间序列，以便从使用labelmap发现的元数据标签中创建标签，将__adress__标签替换为Kubernetes API服务器的默认DNS名称。然后我们使用其中一个元数据标签，一个带有节点名称的标签，在API上创建一个新标签__metrics_path__，它将节点名称传递给路径
代码语言：javascript
复制
/api/v1/nodes/${1}/proxy/metrics/cadvisor
  • 这将为作业发现的每个节点抓取所需的时间序列。在我们的配置中也有一个监控节点本身的作业，它暴露了一些Kubernetes节点级指标
  • 我们可以使用这些指标来监控底层容器、Docker守护进程和Kubernetes级节点性能
代码语言：javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval:     15s
      evaluation_interval: 15s

    alerting:
      alertmanagers:
      - kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: alertmanager
          action: keep
        - source_labels: [__meta_kubernetes_namespace]
          regex: monitoring
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          action: keep
          regex: cluster

    rule_files:
      - "/var/prometheus/rules/*_rules.yml"
      - "/var/prometheus/rules/*_alerts.yml"

    scrape_configs:
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        insecure_skip_verify: true
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

    - job_name: 'kubernetes-apiservers'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

书

  • 《Kubernetes:Up and Running》
本文参与 腾讯云自媒体分享计划，分享自微信公众号。
原始发表：2019-12-18，如有侵权请联系 cloudcommunity@tencent.com 删除
node.js
api
https
网络安全
爬虫

本文分享自 yeedomliu 微信公众号，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ，欢迎热爱写作的你一起参与！

node.js
api
https
网络安全
爬虫
评论
登录后参与评论
0 条评论
热度
最新
登录 后参与评论
推荐阅读
LV.
文章
0
获赞
0
目录
  • 第12章 监控Kubernetes
    • 12.1 Kubernetes集群
      • 12.2 在Kubernetes上运行Prometheus
        • 12.3 监控Kubernetes
          • 12.4 监控Kubernetes节点
            • 12.4.1 Node Exporter DaemonSet
            • 12.4.2 Node Exporter服务
            • 12.4.3 部署Node Exporter
            • 12.4.4 Node Exporter作业
            • 12.4.5 Node Exporter规则
          • 12.5 Kubernetes
            • 12.5.1 Kube-state-metrics
            • 12.5.2 Kube API
            • 12.5.3 cAdvisor和节点
          • 书
          相关产品与服务
          Prometheus 监控服务
          Prometheus 监控服务（TencentCloud Managed Service for Prometheus，TMP）是基于开源 Prometheus 构建的高可用、全托管的服务，与腾讯云容器服务（TKE）高度集成，兼容开源生态丰富多样的应用组件，结合腾讯云可观测平台-告警管理和 Prometheus Alertmanager 能力，为您提供免搭建的高效运维能力，减少开发及运维成本。
          产品介绍产品文档
          精选特惠 用云无忧
          领券

          腾讯云开发者

          扫码关注腾讯云开发者

          扫码关注腾讯云开发者

          领取腾讯云代金券

          热门产品

          热门推荐

          更多推荐

          Copyright © 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

          深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

          腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

          问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档

          Copyright © 2013 - 2024 Tencent Cloud.

          All Rights Reserved. 腾讯云 版权所有

          登录 后参与评论