kubernetes init
流程kubeconfig
文件manifest
文件MasterConfiguration
Master
标志TLS
的安全引导相关的配置DNS
和kube-porxy
插件kubeadm init pre-flight check:
kubeadm
版本要与安装的kubernetes
版本的比对检查kubernetes
安装的系统需求检查root
>,主机,端口,swap
,工具等kubeadm init
生成私钥与证书:
目录在
/etc/kubernetes/pki
下
-rw-r--r-- 1 root root 1224 Oct 12 11:18 apiserver.crt
-rw-r--r-- 1 root root 1094 Oct 12 11:18 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Oct 12 11:18 apiserver-etcd-client.key
-rw------- 1 root root 1675 Oct 12 11:18 apiserver.key
-rw-r--r-- 1 root root 1099 Oct 12 11:18 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Oct 12 11:18 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 Oct 12 11:18 ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 ca.key
drwxr-xr-x 2 root root 4096 Oct 12 11:18 etcd
-rw-r--r-- 1 root root 1025 Oct 12 11:18 front-proxy-ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 front-proxy-ca.key
-rw-r--r-- 1 root root 1050 Oct 12 11:18 front-proxy-client.crt
-rw------- 1 root root 1679 Oct 12 11:18 front-proxy-client.key
-rw------- 1 root root 1679 Oct 12 11:18 sa.key
-rw------- 1 root root 451 Oct 12 11:18 sa.pub
CA
,生成ca.key
与ca.crt
查看公钥证书
$ openssl x509 -in ca.crt -noout -text
--------------------------------------------
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Oct 12 03:18:52 2018 GMT
Not After : Oct 9 03:18:52 2028 GMT
Subject: CN=kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c9:b3:b3:4e:db:bb:a7:be:d1:7e:97:8c:f8:06:
a5:df:04:93:72:32:96:b5:f9:f1:16:20:87:61:10:
44:b3:33:86:16:24:97:bf:4d:ac:04:5f:f9:c9:ec:
61:9d:6c:ce:ad:97:2d:a2:6b:03:e7:b8:89:04:72:
9a:91:2a:da:31:52:8e:f9:86:f3:e4:96:27:56:dd:
e3:8a:84:4e:11:9e:de:0b:c2:2c:73:cd:fd:1f:66:
bf:89:12:38:d5:22:b2:f2:0d:4e:97:e1:59:8c:3d:
8c:df:53:74:d4:2c:84:9f:11:55:84:b7:16:6f:44:
b9:f1:fd:82:fa:67:1c:08:d2:3c:da:01:0f:d7:a4:
4b:85:01:3e:d6:79:dc:96:21:e9:67:b7:0f:f4:bc:
ad:1a:84:71:20:e9:e6:81:f6:a1:8b:26:6b:63:85:
8a:23:f3:f9:6e:bd:ca:28:9f:5e:fe:dd:01:78:53:
0b:fd:01:e9:3a:13:54:5a:32:50:c4:6c:1e:09:4a:
96:33:20:a7:71:03:7a:e9:6b:d8:06:a7:16:86:d8:
cb:15:85:0d:d4:3b:c4:27:69:b0:d8:59:25:b3:b0:
df:60:d5:91:ed:b3:53:77:7a:7b:51:2a:f5:54:56:
db:75:e3:2c:c4:9f:a0:b4:99:b3:da:55:d1:f7:1c:
d8:11
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
02:fb:42:37:10:9d:fb:db:e0:bc:9e:d9:5f:fe:b9:c1:a8:e2:
00:06:85:df:41:25:9b:4f:63:ac:ae:61:9a:5e:0c:62:d6:b8:
24:94:66:ca:c8:bc:33:3e:fe:a8:7b:50:4f:92:86:32:dc:f1:
55:23:97:ed:6e:d9:44:ee:f7:50:01:99:0d:ae:be:c6:9e:16:
76:7c:1d:f5:a0:18:3b:7b:fa:7c:af:90:1b:a3:6c:2a:2b:2e:
aa:c1:ad:fe:82:08:cd:35:47:91:9d:ff:4a:7b:d1:c3:2c:7e:
59:1a:25:d9:77:43:f3:a3:0f:88:43:94:9a:d7:21:4b:01:70:
37:b3:11:9d:6b:58:98:0d:41:f8:d9:64:39:99:bb:45:b7:5d:
f9:46:cf:b3:38:cd:ca:f0:28:0c:68:2d:95:61:97:1f:af:b9:
e7:b1:62:a8:e2:13:83:b0:de:08:7b:6f:7a:5a:ef:1f:9f:bb:
12:ed:e4:5b:8d:49:8e:44:3a:d0:af:36:ef:00:25:1c:bd:47:
12:98:e1:4c:27:eb:ac:48:90:1c:2b:1f:f1:8c:c7:1a:28:b6:
b7:e0:a7:9e:66:19:45:b9:e2:89:9c:14:b4:ac:6d:ba:a8:e8:
f0:b1:c9:52:c2:50:6c:b7:cd:a7:3c:dd:df:d5:0b:16:e2:59:
68:f0:14:c8
Serial Number: 0 (0x0)
表示ca
证书是第一个证书,所以序列号是0Signature Algorithm
表示使用的是sha256
的非对称加密算法Subject
表示comment name
X509v3 Key Usage
表示证书的用途,该证书的用途是数字签名、秘钥加密、证书的签发CA:TRUE
,表明了这个是CA的公钥证书apiserver
的私钥与公钥证书查看公钥证书
$ openssl x509 -in apiserver.crt -noout -text
--------------------------------------------------
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3463380858865092747 (0x3010668a968a888b)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Oct 12 03:18:52 2018 GMT
Not After : Oct 12 03:18:52 2019 GMT
Subject: CN=kube-apiserver
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c2:fb:f7:42:32:e4:74:fb:cc:5a:b8:5d:6e:6f:
74:00:c1:3e:21:e4:4c:76:0e:d2:2d:ea:9f:90:9a:
10:6a:57:a6:d1:d6:24:1c:90:6e:22:65:8c:67:cd:
6a:67:35:0b:4e:50:17:3e:51:05:61:ad:a5:f0:2c:
1a:6a:bd:ac:79:a4:57:da:f6:a4:35:51:8b:5e:04:
1c:03:98:d0:b5:88:87:91:54:cd:15:d6:5a:e4:7f:
38:fa:ee:01:9d:8f:2a:f0:ac:88:5a:a5:8b:6e:ad:
74:d5:43:81:2d:44:01:0b:5a:14:01:03:9e:99:d6:
82:d5:55:7f:40:80:16:e3:33:6c:d8:a7:8e:2d:e9:
7a:ee:66:d0:3d:52:cb:66:ff:f4:a7:a3:a0:5a:db:
b7:38:e3:1b:b3:8e:99:31:d0:bb:7e:92:8d:9d:b2:
df:5e:62:3e:eb:b9:16:3b:14:dc:db:d5:cb:41:05:
e8:c9:cf:1b:75:ba:ba:5f:99:a7:13:90:36:0b:ac:
f1:1c:99:82:c5:4c:b1:3f:ff:04:be:dc:ee:19:c6:
db:4e:16:3e:68:b7:44:78:c2:4c:76:f8:8b:58:8b:
b2:8f:c4:24:6e:d6:64:d1:2a:84:5a:a7:06:6b:95:
e1:94:dd:6f:0c:83:48:32:1c:17:51:50:52:a3:b8:
f0:01
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:k8s1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:xxxxx, IP Address:xxxxx
Signature Algorithm: sha256WithRSAEncryption
17:8c:7e:17:6c:ac:6d:d4:48:f6:8d:81:6b:67:fa:a1:a3:8d:
9f:2c:7b:ed:d6:4c:07:15:96:e5:37:3f:dd:88:30:6a:1d:3a:
52:40:2d:c3:fd:ae:5e:23:d1:d9:8b:f6:56:21:07:e4:98:a6:
04:53:37:93:fb:21:75:3f:fa:3e:23:bb:29:81:9b:ef:7b:ea:
9f:df:5a:5a:22:da:b4:31:4b:b1:f0:bc:ee:63:69:2e:93:f5:
a2:3a:4e:4f:bb:17:5f:f1:87:08:84:ee:59:84:36:ea:6a:b8:
d7:29:db:1d:45:3b:c8:34:f2:29:4d:36:d2:6e:a3:43:63:13:
88:c5:e2:27:78:ed:91:b9:67:81:5d:a6:93:c9:99:25:a5:33:
b7:c5:5f:a8:03:ce:b0:29:55:4f:de:97:3b:75:31:30:9d:58:
75:a1:00:02:5b:c7:41:f9:ac:81:7f:4c:e3:a3:5c:22:7b:7c:
41:25:92:2a:c9:71:c2:90:18:65:48:10:81:8e:c9:34:69:60:
61:a1:4e:4b:cc:6d:36:af:05:01:96:e2:d2:a8:20:60:22:60:
bb:56:bc:e0:11:d9:5b:c5:ec:bd:58:9a:34:1f:95:99:61:c1:
fa:1b:4b:47:1d:68:97:dd:23:3e:42:5c:98:b6:21:8f:96:5d:
52:8c:d9:84
xxx
表示的是我公网IP,这个证书的用于多种域名以及IP地址X509v3 Key Usage
证书用途只有,证书签名,秘钥加密,没有签署证书的用途apiserver
访问kubelet
使用的客户端私钥与证书该证书是
apiserver-kubelet-client.crt
,没有什么特殊的地方,就不展示了
services account
需要的sa.key
与sa.pub
sa.key
用于对account
的token
进行数字签名sa.pub
是key
对应的公钥文件Etcd
相关的私钥与数字证书apiserver
,其他都是通过apiserver
的api
来获取的apiserver
与etcd
之间的安全通道,会生成apiserver
访问etcd
公钥与私钥证书:apiserver-etcd-client.crt
与apiserver-etcd-client.key
etcd
目录下,还有一堆证书-rw-r--r-- 1 root root 1025 Oct 12 11:18 ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 ca.key
-rw-r--r-- 1 root root 1099 Oct 12 11:18 healthcheck-client.crt
-rw------- 1 root root 1675 Oct 12 11:18 healthcheck-client.key
-rw-r--r-- 1 root root 1090 Oct 12 11:18 peer.crt
-rw------- 1 root root 1675 Oct 12 11:18 peer.key
-rw-r--r-- 1 root root 1078 Oct 12 11:18 server.crt
-rw------- 1 root root 1679 Oct 12 11:18 server.key
可以看到也有
CA
证书,那么外面的apiserver-etcd-client.crt
是由那个CA
证书签发的呢? 首先跟外面的对比一下
$ openssl verify -CAfile ca.crt ./apiserver-etcd-client.crt
-------------------------------------------------------------
./apiserver-etcd-client.crt: O = system:masters, CN = kube-apiserver-etcd-client
error 7 at 0 depth lookup:certificate signature failure
140364182943384:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:103:
140364182943384:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:705:
140364182943384:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:218:
可以看到是错误的信息,说明不是由外部的
CA
签发的
etcd
里面的签发的$ openssl verify -CAfile etcd/ca.crt ./apiserver-etcd-client.crt
-----------------------------------------------------------------
./apiserver-etcd-client.crt: OK
说明是由
etcd
目录下的CA
签发的
kubeconfig
文件组件kubeconfig
文件:
.kube/config
、/etc/kubernetes/*.conf
与KUBECONFIG
环境变量/etc/kubernetes
目录下的admin.conf
、kubelet.conf
、scheduler.conf
、controller-manager.conf
admin.conf
包含整个集群的最高权限配置,常用来使用KUBECONFIG
环境变量来设置kubectl
的kubeconfig
信息kubelet.conf
被kubelet
所使用,用于访问apiserver
scheduler.conf
用于master
上的kube-scheduler
组件所使用,用于访问apiserver
controller-manager.conf
用于master
上的kube-controller-manager
组件所使用,用于访问apiserver
kubeconfig
配置:包含cluster
、user
、context
信息使用
kubectl
来查看admin.conf
中的内容
$ kubectl config view
-----------------------------------
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://xxxxxx:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
context
是用来绑定cluster
与users
的current-context
是用来确定当前使用的context
kubernetes-admin@kubernetes
定义了,使用kubernetes-admin
这个用户来访问kubernetes
这个集群kubectl
快速切换context
,管理多集群如果
kubeconfig
定义了多个context
,那么可以通过kubectl set context
命令来设定current-context
manifest
文件组件manifest
文件
/etc/kubernetes/manifests
-rw------- 1 root root 1640 Oct 12 11:19 etcd.yaml
-rw------- 1 root root 2599 Oct 12 11:19 kube-apiserver.yaml
-rw------- 1 root root 2053 Oct 12 11:19 kube-controller-manager.yaml
-rw------- 1 root root 978 Oct 12 11:19 kube-scheduler.yaml
都是标准的
pod
文件,每一个对应一个在master
上的控制组件
Static Pod
形式运行的Static Pod
是以节点上的kubelet
来管理的,不通过master
节点的apiserver
来管理,也不管理任何控制器kubelet
自己来监控,当Static Pod
崩溃的时候,kubelet
会自动重启这些Pod
kubelet
上,并且始终运行在同一节点上kubelet
会自动为每一个Static Pod
在kubernetse
的apiserver
上创建一个镜像的pod
apiserver
查询到该server
,并不能管理控制它kubelet
读取manifests
目录并管理各控制平台组件的pod
启动停止kubeadm
依赖kubelet
下载镜像并启动static pod
k8s.gcr.io
上面下载组件镜像由于国内网络原因,不访问外国网站,是从那上面下载不下来的,解决方法是下载国内的一些镜像,然后将名称更改为所要用的名称
kubeadm
会一直探测并等待localhost:6443/healthz
服务返回成功配置文件存放在
manifests
文件夹下的kube-apiserver.yaml
中
$ cat manifests/kube-apiserver.yaml
-------------------------------------------
... ...
livenessProbe:
failureThreshold: 8
httpGet:
host: xxxxxx
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
... ...
ok
了,可以通过如下命令来查看$ kubectl get pods -n kube-system -o wide
DNS
与kube-proxy
插件安装Addons
DaemonSet
方式部署kube-proxy
查看以
DaemonSet
方式安装的kube-proxy
$ kubectl get daemonset -n kube-system
----------------------------------------------
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-proxy 3 3 3 3 3 <none> 4d
weave-net 3 3 3 3 3 <none> 4d
kube-dns
(也可以使用CoreDNS
代替)DNS
插件,会显示pending
状态,直到cluster
网络就绪可以通过安装
weave-net
来安装网络插件,使得DNS
处于running
状态
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
若是网络IP冲突,可以将该文件下载下来,查找到weave的容器(搜索:containers),写入
- name: IPALLOC_RANG
value: xxxxxx/xxxxxxx
## xxx表示自己在init的时候,给定的IP地址
$ kubectl get pods -n kube-system -o wide
原dns那个也running了
本文作者为olei,转载请注明。