查看cs状态：发现是有问题的 [root@master1 ~]#kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME..."http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused scheduler...Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection...解决思路： 注释掉/etc/kubernetes/manifests下的kube-controller-manager.yaml和kube-scheduler.yaml的- – port=0 确认kube-scheduler...和10252 [root@k8s-master ~]# netstat -tunlp | grep 10251 tcp6 0 0 :::10251 :
报错： 使用Kubeadm安装的K8s集群获取kube-scheduler和kube-controller-manager组件状态异常，基本上都会出现这个问题。...[root@k8s-master ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME..."http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager...tier: control-plane name: kube-scheduler namespace: kube-system spec: containers: - command..."http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused etcd-0
ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252:...connect: connection refused scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial...tcp 127.0.0.1:10251: connect: connection refused etcd-0 Healthy {"health":"true"}...0 127.0.0.1:10257 0.0.0.0:* LISTEN 12833/kube-controll tcp 0 0...0 :::10251 :::* LISTEN 12385/kube-schedule tcp6 0 0
故障现象 查看状态存在不健康组件 [root@k8s-master ~]#kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19...ERROR scheduler Unhealthy Get..."http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager...Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused...manifests]#systemctl restart kubelet.service 再次查看cs状态 [root@k8s-master /etc/kubernetes/manifests]#kubectl get
/cluster/get-kube-binaries.sh ......://sz-pg-oam-docker-test-001.tendcloud.com:8080" KUBE_MASTER="--master=http://192.168.1.121:8080" 该配置文件同时被...100 --audit-log-path=/var/lib/audit.log --event-ttl=1h" --authorization-mode=RBAC 指定在安全端口使用 RBAC 授权模式，拒绝未通过授权的请求...http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager...http://172.20.0.113:2379/health: malformed HTTP response "\x15\x03\x01\x00\x02\x02" etcd-0
18.启动之后如下图及安装成功： 19.插入一个小问题： kubeadm安装的k8s集群获取kube-scheduler和kube-controller-manager组件状态异常，一般都会有...ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252:...connect: connection refused scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial...tcp 127.0.0.1:10251: connect: connection refused etcd-0 Healthy {"health":"true"}...kubectl get pods kubectl get pods -o wide kubectl get pods --show-labels #查看pod的标签： kubectl describe
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect...: connection refused controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp...127.0.0.1:10252: connect: connection refused 原因：主要是因为托管集群部署架构导致不能通过127.0.0.1 连接到相关组件，实际上组件状态是正常的，可以忽略此告警...被拒绝）。...DNS 解析超时/失败问题 查看请求解析容器的 /etc/resolv.conf 配置信息，确保配置无误。 1.
--kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver； --leader-elect=true：集群运行模式，启用选举功能...|grep Active" 6 done 2.3 查看输出的 metrics kube-scheduler 监听 10251 和 10251 端口： 10251：接收 http 请求，非安全端口...两个接口都对外提供 /metrics 和 /healthz 的访问。 1 [root@k8smaster01 ~]# sudo netstat -lnpt |grep kube-sch ?...1 [root@k8smaster01 ~]# curl -s http://127.0.0.1:10251/metrics |head 2 [root@k8smaster01 ~]# curl -...2.4 查看当前leader 1 [root@k8smaster01 ~]# kubectl get endpoints kube-scheduler --namespace=kube-system
port: 10251 targetPort: 10251 protocol: TCP复制代码 10251是kube-scheduler组件 metrics 数据所在的端口，10252...我们可以看到现在已经发现了 target，但是抓取数据结果出错了，这个错误是因为我们集群是使用 kubeadm 搭建的，其中 kube-scheduler 默认是绑定在127.0.0.1上面的，而上面我们这个地方是想通过节点的...IP 去访问，所以访问被拒绝了，我们只要把 kube-scheduler 绑定的地址更改成0.0.0.0即可满足要求，由于 kube-scheduler 是以静态 Pod 的形式运行在集群中的，所以我们只需要更改静态...: type: ClusterIP clusterIP: None ports: - name: http-metrics port: 10251 targetPort: 10251...port: 10251 protocol: TCP复制代码（2）、定义ServiceMonitor prometheus-serviceMonitorKubeScheduler.yamlapiVersion
ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252:...connect: connection refused scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial...tcp 127.0.0.1:10251: connect: connection refused etcd-0 Healthy {"health":"true"}...ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251:...kube-scheduler监听10251和10259端口： 10251：接收 http 请求，非安全端口，不需要认证授权； 10259：接收 https 请求，安全端口，需要认证授权； 两个接口都对外提供
: type: ClusterIP clusterIP: None ports: - name: http-metrics port: 10251 targetPort:...10251 protocol: TCP --- apiVersion: v1 kind: Endpoints metadata: labels: k8s-app: kube-scheduler...地址 ports: - name: http-metrics port: 10251 protocol: TCP 坑二 访问prometheus server的web页面发现即使创建了...svc和注入对应ep的信息在target页面还是被prometheus server请求被拒绝 修改 kube-controller-manager 配置文件 把 --address=127.0.0.1...换成 --address=0.0.0.0 修改 kube-scheduler 配置文件 把 --address=127.0.0.1 换成 --address=0.0.0.0 修改完后重启这两个服务 坑三
leader选举功能（--leader-elect=true），使得3个controller-manager和scheduler都分别只有一个是leader，leader处于正常工作状态，当leader失败...--etcd-servers=http://127.0.0.1:4001 --cloud-provider=gce --admission-control=NamespaceLifecycle.../bin/kube-scheduler --master=127.0.0.1:8080 --v=2 --leader-elect=true 1>>/var/log/kube-scheduler.log...2>&1 livenessProbe: httpGet: path: /healthz port: 10251 initialDelaySeconds...- http://${NODE_IP}:4001 - --listen-client-urls - http://127.0.0.1:4001 - --data-dir
VM-1-6-centos VM-1-6-centos 127.0.0.1 localhost.localdomain localhost 127.0.0.1 localhost4.localdomain4...br_netfilter后，再执行 sysctl -p生效 配置相关下载源以及安装Docker 下载Docker镜像源： wget -O /etc/yum.repos.d/docker-ce.repo http..."http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager...Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused...查看集群所有节点，通过管理节点也就是master节点执行 kubectl get nodes查看： [root@VM-1-7-centos ~]# kubectl get nodes NAME
如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。...：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver； leader-elect=true：集群运行模式，启用选举功能；被选为 leader...l、查看输出的 metrics 注意：以下命令在 kube-scheduler 节点上执行 kube-scheduler 监听 10251 和 10251 端口： 10251：接收 http 请求，非安全端口...[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 work]# curl -s http://172.26.16.249:10251/metrics|head...e、查看当前leader [root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 work~]# kubectl get endpoints kube-scheduler
✅ NodeOOM 监控那些消耗过多内存的进程，尤其是那些消耗大量内存非常快的进程，内核会杀掉它们，防止它们耗尽内存 ✅ NodeExt4Error Ext4 挂载失败 ✅ NodeTaskHung 检查...检查对应存储 ✅ PodDeviceOrResourceBusy 检查对应的目录和 PID ✅ PodFileExists 检查现有文件 ✅ PodTooManyOpenFiles 程序打开的文件/套接字连接数超过系统设置值...docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock....http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused etcd-0...Fatal 2020-11-27T17:56:37+08:00 Get https://192.168.13.8:2379/health: dial tcp 192.168.13.8:
: http-metrics port: 10251 targetPort: 10251 protocol: TCP 10251是kube-scheduler组件 metrics...PORT(S) AGE kube-scheduler ClusterIP 10.102.119.231 10251/TCP 18m 创建完成后...所以访问被拒绝了，我们只要把 kube-scheduler 绑定的地址更改成0.0.0.0即可满足要求，由于 kube-scheduler 是以静态 Pod 的形式运行在集群中的，所以我们只需要更改静态...: 10251 protocol: TCP --- apiVersion: v1 kind: Endpoints metadata: name: kube-scheduler namespace...: - name: http-metrics port: 10251 protocol: TCP （2）、定义ServiceMonitor prometheus-serviceMonitorKubeScheduler.yaml
10251？那我改一下10251试试？（虽然忘了官方从1.17还是哪个版本就默认值开通10259了吧？...type: ClusterIP clusterIP: None ports: - name: http-metrics port: 10251 targetPort:...10251 protocol: TCP --- apiVersion: v1 kind: Endpoints metadata: labels: k8s-app: kube-scheduler...- ip: 10.0.4.38 ports: - name: http-metrics port: 10251 protocol: TCP --- EOF kubectl.../kubernetes.io/serviceaccount/token interval: 30s port: http-metrics scheme: http tlsConfig
Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused....Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused....Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused....Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused....Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
这样的 Service，但是我们系统中根本就没有对应的 Service： $ kubectl get svc -n kube-system -l k8s-app=kube-scheduler No resources...clusterIP: None ports: - name: http-metrics port: 10251 targetPort: 10251 protocol...PORT(S) AGE kube-scheduler ClusterIP None 10251/TCP 16s 可以用同样的方式来修复下...IP 是 127.0.0.1 地址。...connect refused 经过分析发现是 /etc/kubernetes/manifests 下的 kube-controller-manager.yaml 和 kube-scheduler.yaml
://localhost:10256/healthz kube-scheduler: name: kube-scheduler command: - /opt...://localhost:10251/healthz kubelet: name: kubelet command: - /opt/rke/entrypoint.sh...port: 10250 protocol: TCP - address: 127.0.0.1 port: 6443 protocol: TCP - address...: 127.0.0.1 port: 2379 protocol: TCP - address: 127.0.0.1 port: 2380 protocol: TCP...，代码片段展示如下： ...... func (g *genericController) processNextWorkItem() bool { key, quit := g.queue.Get
领取专属 10元无门槛券
手把手带您无忧上云
云服务器
即时通信 IM
ICP备案
对象存储
实时音视频
