Kubernetes全栈架构师（二进制高可用安装k8s集群扩展篇）--学习笔记

原创

郑子铭

修改于 2021-10-13 10:40:33

1.4K0

修改于 2021-10-13 10:40:33

git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes
# ./deploy.sh -s -i 10.96.0.10 | kubectl apply -f -
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created

查看状态

kubectl get po -n kube-system -l k8s-app=kube-dns

状态

NAME                      READY   STATUS    RESTARTS   AGE
coredns-fb4874468-nr5nx   1/1     Running   0          49s

强制删除一直处于Terminating的pod

[root@k8s-master01 ~]# kubectl get po -n kube-system -l k8s-app=kube-dns
NAME                      READY   STATUS        RESTARTS   AGE
coredns-fb4874468-fgs2h   1/1     Terminating   0          6d20h

[root@k8s-master01 ~]# kubectl delete pods coredns-fb4874468-fgs2h --grace-period=0 --force -n kube-system
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "coredns-fb4874468-fgs2h" force deleted

[root@k8s-master01 ~]# kubectl get po -n kube-system -l k8s-app=kube-dns
No resources found in kube-system namespace.

安装Metrics Server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

安装metrics server

cd /root/k8s-ha-install/metrics-server-0.4.x/

kubectl  create -f .

等待metrics server启动然后查看状态

kubectl  top node

节点状态

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   263m         13%    1239Mi          66%       
k8s-master02   213m         10%    1065Mi          57%       
k8s-master03   207m         10%    1050Mi          56%       
k8s-node01     89m          4%     514Mi           27%       
k8s-node02     158m         7%     493Mi           26%

查看pod状态

kubectl  top po -A

pod状态

NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)   
kube-system   calico-kube-controllers-cdd5755b9-4fzg9   3m           18Mi            
kube-system   calico-node-8xg62                         26m          60Mi            
kube-system   calico-node-dczxz                         24m          60Mi            
kube-system   calico-node-gn8ws                         23m          62Mi            
kube-system   calico-node-qmwkd                         26m          60Mi            
kube-system   calico-node-zfw8n                         25m          59Mi            
kube-system   coredns-fb4874468-nr5nx                   3m           10Mi            
kube-system   metrics-server-64c6c494dc-9x727           2m           18Mi

安装dashboard

安装指定版本dashboard
安装最新版dashboard
登录dashboard

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

安装指定版本dashboard

cd /root/k8s-ha-install/dashboard/

kubectl  create -f .

安装最新版dashboard

官方GitHub地址：https://github.com/kubernetes/dashboard

可以在官方dashboard查看到最新版dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

创建管理员用户

vim admin.yaml
# 添加以下内容
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding 
metadata: 
  name: admin-user
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

执行

kubectl apply -f admin.yaml -n kube-system

登录dashboard

在谷歌浏览器（Chrome）启动文件中加入启动参数，用于解决无法访问Dashboard的问题，因为使用的证书是自签名（属性->快捷方式->目标，粘贴到最后）

 --test-type --ignore-certificate-errors

更改dashboard的svc为NodePort：

kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

修改 type: ClusterIP 为 type:NodePort

修改完成之后会暴露一个端口号，查看端口号：

kubectl get svc kubernetes-dashboard -n kubernetes-dashboard

端口号

NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.108.217.183   <none>        443:31874/TCP   9m37s

根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard：访问Dashboard：https://192.168.232.236:31874（请更改18282为自己的端口），选择登录方式为令牌（即token方式）

查看token值：

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

token值

Name:         admin-user-token-9c4tz
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: d1f2e528-0ef8-4c6b-a384-a18fbca6bc54

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1411 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlNCbEdFa1RQZElhbTBRb29aTTNCTUE1dTJ2enBCeGZxMWJwbmpfZHBXdkEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTljNHR6Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkMWYyZTUyOC0wZWY4LTRjNmItYTM4NC1hMThmYmNhNmJjNTQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.KFH5ed0kJEaU1HSpxkitJxqKJGnSNAWogNSGjGn1wEh7R9zKYkAfNLES6Vl3GU9jvxBCEZW415ZFILr96kpgl_88mD-K-AMgQxKLdpghYDx_CnsLtI6e8rLTNkaPS2Uo3sYAy9U280Niop14Yzuar5FQ3AfSbeXGcF_9Jrgyeh5XWPA0h69Au8pUEOkVdpADmuIaFSqfTnmkOSdGqCgFb_QsUqvjo4ifIxKnN6uW8wfR1s4esWkPq569xhCINaUY6g3rnT1jfVTU2XmrURrKOVok0OfSmtXTKCSs2jliEdmx7qEFTrw2KCPnTfORUtTnmdZ2ZnGGx9Fvf_hGaKk1FQ

二进制高可用集群可用性验证

安装busybox

[root@k8s-master01 ~]# cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

查看状态

[root@k8s-master01 ~]# kubectl get po
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          29s

Pod必须能解析Service
Pod必须能解析跨namespace的Service
每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
Pod和Pod之前要能通（同namespace能通信、跨namespace能通信、跨机器能通信）

集群安装成功后默认的kubernetes server

[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d22h

Pod必须能解析Service

[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

Pod必须能解析跨namespace的Service

[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53（发送键输入到所有的会话）

[root@k8s-master01 ~]# yum install telnet -y

[root@k8s-master01 ~]# telnet 10.96.0.1 443
Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.

[root@k8s-master01 ~]# kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns         ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   53m
metrics-server   ClusterIP   10.107.95.145   <none>        443/TCP                  4h33m
You have new mail in /var/spool/mail/root

[root@k8s-master01 ~]# telnet 10.96.0.10 53
Trying 10.96.0.10...
Connected to 10.96.0.10.
Escape character is '^]'.

[root@k8s-master01 ~]# curl 10.96.0.10:53
curl: (52) Empty reply from server

Pod和Pod之前要能通（同namespace能通信、跨namespace能通信、跨机器能通信）（取消发送键输入到所有的会话）

[root@k8s-master01 ~]# kubectl get po -n kube-system -owide
NAME                                      READY   STATUS    RESTARTS        AGE     IP                NODE           NOMINATED NODE   READINESS GATES
calico-kube-controllers-cdd5755b9-4fzg9   1/1     Running   0               4h59m   192.168.232.131   k8s-node01     <none>           <none>
calico-node-8xg62                         1/1     Running   0               4h59m   192.168.232.129   k8s-master02   <none>           <none>
calico-node-dczxz                         1/1     Running   0               4h59m   192.168.232.131   k8s-node01     <none>           <none>
calico-node-gn8ws                         1/1     Running   0               4h59m   192.168.232.128   k8s-master01   <none>           <none>
calico-node-qmwkd                         1/1     Running   0               4h59m   192.168.232.130   k8s-master03   <none>           <none>
calico-node-zfw8n                         1/1     Running   2 (4h59m ago)   4h59m   192.168.232.132   k8s-node02     <none>           <none>
coredns-fb4874468-fgs2h                   1/1     Running   0               56m     172.25.92.66      k8s-master02   <none>           <none>
metrics-server-64c6c494dc-9x727           1/1     Running   0               4h35m   172.27.14.193     k8s-node02     <none>           <none>

# 进入k8s-master02
[root@k8s-master01 ~]# kubectl exec -ti calico-node-8xg62  -n kube-system -- bash
Defaulted container "calico-node" out of: calico-node, install-cni (init), flexvol-driver (init)

# ping k8s-master03
[root@k8s-master02 /]# ping 192.168.232.130
PING 192.168.232.130 (192.168.232.130) 56(84) bytes of data.
64 bytes from 192.168.232.130: icmp_seq=1 ttl=64 time=0.416 ms
64 bytes from 192.168.232.130: icmp_seq=2 ttl=64 time=0.240 ms
64 bytes from 192.168.232.130: icmp_seq=3 ttl=64 time=0.191 ms

# 退出k8s-master02
[root@k8s-master02 /]# exit
exit

[root@k8s-master01 ~]# kubectl get po -owide
NAME      READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
busybox   1/1     Running   0          22m   172.25.92.69   k8s-master02   <none>           <none>

# 进入k8s-master02容器ping k8s-master03
[root@k8s-master01 ~]# kubectl exec -ti busybox -- sh
/ # ping 192.168.232.130
PING 192.168.232.130 (192.168.232.130): 56 data bytes
64 bytes from 192.168.232.130: seq=0 ttl=63 time=0.329 ms
64 bytes from 192.168.232.130: seq=1 ttl=63 time=0.452 ms
64 bytes from 192.168.232.130: seq=2 ttl=63 time=0.675 ms

创建一个带有三个副本的deployment

[root@k8s-master01 ~]# kubectl create deploy nginx --image=nginx --replicas=3
deployment.apps/nginx created

[root@k8s-master01 ~]# kubectl get deploy
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   2/3     3            2           35s

# 查看部署所在的NODE
[root@k8s-master01 ~]# kubectl get po -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
busybox                  1/1     Running   0          28m     172.25.92.69     k8s-master02   <none>           <none>
nginx                    1/1     Running   0          2m37s   172.18.195.4     k8s-master03   <none>           <none>
nginx-6799fc88d8-lbhgm   1/1     Running   0          54s     172.25.244.197   k8s-master01   <none>           <none>
nginx-6799fc88d8-nq2gz   1/1     Running   0          54s     172.17.125.1     k8s-node01     <none>           <none>
nginx-6799fc88d8-tzgz8   1/1     Running   0          54s     172.27.14.194    k8s-node02     <none>           <none>

# 删除
[root@k8s-master01 ~]# kubectl delete deploy nginx
deployment.apps "nginx" deleted

[root@k8s-master01 ~]# kubectl delete po busybox

生产环境k8s集群关键性配置

docker配置
controller-manager配置
kubelet配置
kubelet-conf.yml
安装总结

docker配置

（发送键输入到所有的会话）

vim /etc/docker/daemon.json

{  
    "registry-mirrors": [
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn"
  ],
 "exec-opts": ["native.cgroupdriver=systemd"],
 "max-concurrent-downloads": 10, # 并发下载的线程数
 "max-concurrent-uploads": 5, # 并发上传的线程数
 "log-opts": {
     "max-size": "300m", # 限制日志文件最大容量，超过则分割
     "max-file": "2" # 日志保存最大数量
 },
 "live-restore": true # 更改docker配置之后需要重启docker才能生效，这个参数可以使得重启docker不影响正在运行的容器进程
}

更新配置

systemctl daemon-reload

controller-manager配置

注释新版本已经默认配置，设置证书过期时间

（发送键输入到所有的会话，取消node节点）

vim /usr/lib/systemd/system/kube-controller-manager.service

# --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
--cluster-signing-duration=876000h0m0s \

更新配置

systemctl daemon-reload

systemctl restart kube-controller-manager

kubelet配置

（发送键输入到所有的会话）

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service] 
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig" 
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" 
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1" 
Environment="KUBELET_EXTRA_ARGS=--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    --image-pull-progress-deadline=30m" 
ExecStart= 
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

k8s默认加密方式会被识别为漏洞，需要修改加密方式

--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

kubelet下载镜像的deadline，避免重复循环加载

--image-pull-progress-deadline=30m

kubelet-conf.yml

vim /etc/kubernetes/kubelet-conf.yml

# 添加如下配置
rotateServerCertificates: true
allowedUnsafeSysctls: # 允许修改内核，才能修改内核参数，比如修改最大并发量，但是涉及到安全问题，所以按需配置
 - "net.core*"
 - "net.ipv4.*"
kubeReserved: # 预留资源，生产环境需要设置高一点，预留足够资源
  cpu: "10m"
  memory: 10Mi
  ephemeral-storage: 10Mi
systemReserved:
  cpu: "10m"
  memory: 20Mi
  ephemeral-storage: 1Gi

更新配置

systemctl daemon-reload

systemctl restart kubelet

查看日志

tail -f /var/log/messages

验证

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE    VERSION
k8s-master01   Ready    <none>   7h3m   v1.22.0-beta.1
k8s-master02   Ready    <none>   7h3m   v1.22.0-beta.1
k8s-master03   Ready    <none>   7h3m   v1.22.0-beta.1
k8s-node01     Ready    <none>   7h3m   v1.22.0-beta.1
k8s-node02     Ready    <none>   7h3m   v1.22.0-beta.1

安装总结

kubeadm
二进制
自动化安装
安装需要注意的细节

自动化安装（Ansible）

Master节点安装不需要写自动化。
添加Node节点，playbook。

安装需要注意的细节

上面的细节配置
生产环境中etcd一定要和系统盘分开，一定要用ssd硬盘。
Docker数据盘也要和系统盘分开，有条件的话可以使用ssd硬盘

Bootstrapping: Kubelet启动过程

Bootstrapping：自动为node节点颁发证书

二进制高可用安装生成证书的时候，我们为每个k8s组件都生成了一个kubeconfig文件，比如controller-manager.kubeconfig，配置中保存了apiserver的一些信息，以及它去连接apiserver的证书

kube-controller-manager.service指定了kubeconfig，组件启动的时候会通过config文件去连接apiserver进行认证，通讯

[root@k8s-master01 kubernetes]# vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --root-ca-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \ # 指定了kubeconfig
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --pod-eviction-timeout=2m0s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=172.16.0.0/12 \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --node-cidr-mask-size=24

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

kube-controller-manager.service和kube-scheduler.service是我们自己颁发证书，但是kubelet不建议通过这种方式，建议使用TLS BootStrapping，它会自动地为node节点的kubelet颁发证书，那么它是如何生成的呢？

使用node02节点举例

[root@k8s-node02 ~]# cd /etc/kubernetes
[root@k8s-node02 kubernetes]# ls
bootstrap-kubelet.kubeconfig  kubelet-conf.yml  kubelet.kubeconfig  kube-proxy.conf  kube-proxy.kubeconfig  manifests  pki

当我们配置一个节点的时候，节点上面是没有kubelet.kubeconfig文件的，它是通过bootstrap-kubelet.kubeconfig与apiserver进行交互，然后自动申请了一个kubelet.kubeconfig文件，kubelet启动的时候如果缺少kubelet.kubeconfig文件，就会这样申请

删除kubelet.kubeconfig

[root@k8s-node02 kubernetes]# rm -rf kubelet.kubeconfig

查看kubelet配置文件

[root@k8s-node02 kubernetes]# cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    --image-pull-progress-deadline=30m" "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

在配置中可以看到指定了bootstrap-kubeconfig文件

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

指定了kubeconfig文件，但是这个文件是没有的，kubelet启动的时候会通过前面描述的方式申请一个新的kubelet.kubeconfig

--kubeconfig=/etc/kubernetes/kubelet.kubeconfig

重启kubelet，可以看到生成了kubelet.kubeconfig证书，生成之后就可以和apiserver进行交互

[root@k8s-node02 kubernetes]# systemctl restart kubelet
[root@k8s-node02 kubernetes]# ls
bootstrap-kubelet.kubeconfig  kubelet-conf.yml  kubelet.kubeconfig  kube-proxy.conf  kube-proxy.kubeconfig  manifests  pki

TLS BootStrapping 官方文档：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#initialization-process

在一个k8s集群中，工作节点的组件kubelet和kube-proxy需要连接master节点的组件，尤其是apiserver。为了确保连接是私有的，强烈建议为每一个客户端配置一个TLS证书在所有的节点上

kubelet启动过程

查找kubeconfig文件，文件一般位于/etc/kubernetes/kubelet.kubeconfig
从kubeconfig文件中检索APIServer的URL和证书
然后去和APIServer进行交互

查看证书

[root@k8s-node02 kubernetes]# more kubelet.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6
RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10
CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh
bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY
TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO
QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr
WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2
ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o
ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY
WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz
bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox
anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.232.236:8443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

查看kubelet.kubeconfig证书有效期：

[root@k8s-node02 kubernetes]# echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6
RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10
CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh
bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY
TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO
QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr
WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2
ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o
ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY
WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz
bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox
anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
U5EIENFUlRJRklDQVRFLS0tLS0K" | base64 --decode >/tmp/1

然后使用OpenSSL即可查看证书过期时间（100年有效期）：

[root@k8s-node02 kubernetes]# openssl x509 -in /tmp/1 -noout -dates
notBefore=Jul  9 07:35:00 2021 GMT
notAfter=Jun 15 07:35:00 2121 GMT

Bootstrapping: CSR申请和证书颁发原理

官方文档：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#bootstrap-initialization

1、 Kubelet启动

2、 Kubelet查看kubelet.kubeconfig文件，假设没有这个文件

3、 Kubelet会查看本地的bootstrap.kubeconfig

4、 Kubelet读取bootstrap.kubeconfig文件，检索apiserver的url和一个token

5、 Kubelet链接apiserver，使用这个token进行认证

a) Apiserver会识别tokenid，apiserver会查看该tokenid对于的bootstrap的一个secret

secret就是二进制高可用安装集群的时候创建的bootstrap.secret.yaml

如果要修改bootstrap.secret.yaml的token-id和token-secret，需要保证 c8ad9c 字符串一致的，并且位数是一样的。还要保证上个命令的黄色字体：c8ad9c.2e4d610cf3e7426e与你修改的字符串要一致

[root@k8s-master01 kubernetes]# more bootstrap-kubelet.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6
RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10
CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh
bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY
TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO
QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr
WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2
ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o
ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY
WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz
bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox
anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.232.236:8443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: tls-bootstrap-token-user
  name: tls-bootstrap-token-user@kubernetes
current-context: tls-bootstrap-token-user@kubernetes
kind: Config
preferences: {}
users:
- name: tls-bootstrap-token-user
  user:
    token: c8ad9c.2e4d610cf3e7426e

根据后缀c8ad9c找到token

[root@k8s-master01 kubernetes]# kubectl get secret -n kube-system
NAME                                             TYPE                                  DATA   AGE
bootstrap-token-c8ad9c                           bootstrap.kubernetes.io/token         6      3d18h

读取secret中的内容，得到token-id，token-secret

[root@k8s-master01 kubernetes]# kubectl get secret -n kube-system bootstrap-token-c8ad9c -n kube-system -oyaml
apiVersion: v1
data:
  auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz
  description: VGhlIGRlZmF1bHQgYm9vdHN0cmFwIHRva2VuIGdlbmVyYXRlZCBieSAna3ViZWxldCAnLg==
  token-id: YzhhZDlj
  token-secret: MmU0ZDYxMGNmM2U3NDI2ZQ==
  usage-bootstrap-authentication: dHJ1ZQ==
  usage-bootstrap-signing: dHJ1ZQ==
kind: Secret
metadata:
  creationTimestamp: "2021-07-09T09:54:36Z"
  name: bootstrap-token-c8ad9c
  namespace: kube-system
  resourceVersion: "1408"
  uid: 9b77f873-d449-4ab2-aed7-fce9c32bdb21
type: bootstrap.kubernetes.io/token

解密token-id，与配置文件kubectl config中的token=c8ad9c.2e4d610cf3e7426e的前缀一致

[root@k8s-master01 kubernetes]# echo "YzhhZDlj" | base64 -d
c8ad9c

解密token-secret，与配置文件kubectl config中的token=c8ad9c.2e4d610cf3e7426e的后缀一致

[root@k8s-master01 kubernetes]# echo "MmU0ZDYxMGNmM2U3NDI2ZQ==" | base64 -d
2e4d610cf3e7426e

验证通过之后才会进入下一步

b) 找个这个secret中的一个字段auth-extra-groups，apiserver把这个token识别成一个username，名称是system:bootstrap:，属于system:bootstrappers这个组，这个组具有申请csr的权限，该组的权限绑定在一个叫system:node-bootstrapper的clusterrole

解密auth-extra-groups查询组名

[root@k8s-master01 kubernetes]# echo "c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz" | base64 -d
system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress

clusterrole是k8s集群级别的权限控制，它作用整个k8s集群
clusterrolebinding是集群权限的绑定，它可以帮某个clusterrole绑定到一个用户、组或者seviceaccount

查看clusterrole

[root@k8s-master01 kubernetes]# kubectl get clusterrole
NAME                                                                   CREATED AT
system:node-bootstrapper                                               2021-07-09T09:34:55Z

查看system:node-bootstrapper

[root@k8s-master01 kubernetes]# kubectl get clusterrole system:node-bootstrapper -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2021-07-09T09:34:55Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node-bootstrapper
  resourceVersion: "96"
  uid: b44bed52-fce4-4cf3-b3b1-38f887749d70
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests # 定义了一个csr权限
  verbs:
  - create
  - get
  - list
  - watch

查看clusterrolebinding

[root@k8s-master01 kubernetes]# kubectl get clusterrolebinding kubelet-bootstrap -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2021-07-09T09:54:36Z"
  name: kubelet-bootstrap
  resourceVersion: "1409"
  uid: 40de9025-a8f6-41c2-be85-763daca80bb2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token

可以看到clusterrolebinding把一个名为system:node-bootstrapper的ClusterRole绑定到一个名为system:bootstrappers:default-node-token的Group

查看Group是否一致，解密auth-extra-groups查询组名

[root@k8s-master01 kubernetes]# echo "c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz" | base64 -d
system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress

c) CSR：相当于一个申请表，可以拿着这个申请表去申请我们的证书。

6、经过上面的认证，kubelet就有了一个创建和检索CSR的权限

7、 Kubelet为自己创建一个CSR，名称为kubernetes.io/kube-apiserver-client-kubelet

8、 CSR被允许有两种方式：

a) K8s管理员使用kubectl手动的颁发证书

b) 如果配置了相关权限，kube-controller-manager会自动同意。

i. Controller-manager有一个CSRApprovingController。他会校验kubelet发来的csr的username和group是否有创建csr的权限，而且还要验证签发着是否是kubernetes.io/kube-apiserver-client-kubelet

ii. Controller-manager同意CSR请求

Bootstrapping: 证书自动续期原理

9、 CSR被同意后，controller-manager创建kubelet的证书文件

10、 Controller-manager将证书更新至csr的status字段

11、 Kubelet从apiserver获取证书

12、 Kubelet从获取到的key和证书文件创建kubelet.kubeconfig

13、 Kubelet启动完成并正常工作

14、可选：如果配置了自动续期，kubelet会在证书文件过期的时候利用之前的kubeconfig文件去申请一个新的证书，相当于续约。

15、新的证书被同意或签发，取决于我们的配置

查看node-autoapprove-certificate-rotation

[root@k8s-master01 kubernetes]# kubectl get clusterrolebinding node-autoapprove-certificate-rotation -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2021-07-09T09:54:36Z"
  name: node-autoapprove-certificate-rotation
  resourceVersion: "1411"
  uid: d7c618c8-0860-4d03-949f-4e2bbd33659a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

查看名为system:certificates.k8s.io:certificatesigningrequests:selfnodeclient的ClusterRole

[root@k8s-master01 kubernetes]# kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:selfnodeclient -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2021-07-09T09:34:55Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  resourceVersion: "103"
  uid: d5215b54-8dd4-47ba-9e6e-bf664eb56dce
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeclient # 自动续期权限
  verbs:
  - create

a) Kubelet创建的CSR是属于一个组织：system:nodes

b) CN（比如域名）：system:nodes:主机名

课程链接（私信领取福利）

http://www.kubeasy.com/

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes