K8S(V1.10.1)高可用集群超详细版本(包含Dashboard、Rancher)

互联网-小阿宇

发布于 2022-11-21 18:48:47

3020

发布于 2022-11-21 18:48:47

IP	主机名	CPU	运行内存	备注
192.168.1.10	node01	2	4	Master and etcd
192.168.1.20	node02	2	4	Master and etcd
192.168.1.30	node03	2	4	Master and etcd
192.168.1.40	node04	1	2	node
192.168.1.50	node05	1	2	node
192.168.1.60	node06	1	2	node

软件	版本
kubernetes等组件	V1.10.1
docker	V1.13.1

①环境初始化

分别在6台主机设置主机名称

hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname node04
hostnamectl set-hostname node05
hostnamectl set-hostname node06

配置主机映射

echo '192.168.1.10 node01
192.168.1.20 node02
192.168.1.30 node03
192.168.1.40 node04
192.168.1.50 node05
192.168.1.60 node06' >> /etc/hosts

node01上执行ssh免密码登陆配置

ssh-keygen  #一路回车即可
ssh-copy-id  -i node01
ssh-copy-id  -i node02
ssh-copy-id  -i node03
ssh-copy-id  -i node04
ssh-copy-id  -i node05
ssh-copy-id  -i node06

六台主机配置、停防火墙、关闭Swap、关闭Selinux、设置内核、安装依赖包、配置ntp（配置完后建议重启一次）

systemctl stop firewalld
systemctl disable firewalld

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce  0 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  

modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge


yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl 

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
 
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf
-----------------------------------------------------------------------------------------------------------------------
=======================================================================================================================
-----------------------------------------------------------------------------------------------------------------------
重启过后建议在执行一次
sysctl -p /etc/sysctl.d/k8s.conf
如果报错：
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
再次执行以下命令：
modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

--------------------------------------------------------------------------------
修改iptables的另一种方法 区别：
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
1.临时开启，（写入内存，在内存中开启）
echo "1" > /proc/sys/net/ipv4/ip_forward

2.永久开启，（写入内核）
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf 
sysctl -p

②创建etcd证书(node01上执行即可)

设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

创建 CA 配置文件（下面配置的IP为etc节点的IP）

mkdir /root/ssl
cd /root/ssl
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.10",
    "192.168.1.20",
    "192.168.1.30"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

node01分发etcd证书到node02、node03上面

mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n node02 "mkdir -p /etc/etcd/ssl && exit"
ssh -n node03 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem node02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node03:/etc/etcd/ssl/

安装配置etcd (三主节点）安装etcd

yum install etcd -y
mkdir -p /var/lib/etcd

node01的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node01 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.1.10:2380 \
  --listen-peer-urls https://192.168.1.10:2380 \
  --listen-client-urls https://192.168.1.10:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.10:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node02的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node02 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.1.20:2380 \
  --listen-peer-urls https://192.168.1.20:2380 \
  --listen-client-urls https://192.168.1.20:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.20:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node03的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node03 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.1.30:2380 \
  --listen-peer-urls https://192.168.1.30:2380 \
  --listen-client-urls https://192.168.1.30:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.30:2379 \
  --initial-cluster-token etcd-cluster-0 \
--initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

添加自启动（etc集群最少2个节点才能启动，启动报错看mesages日志）

 mv /etc/systemd/system/etcd.service /usr/lib/systemd/system/
 systemctl daemon-reload
 systemctl enable etcd
 systemctl start etcd
 systemctl status etcd

在三个etcd节点执行一下命令检查

etcdctl --endpoints=https://192.168.1.10:2379,https://192.168.1.20:2379,https://192.168.1.30:2379 \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

③所有节点安装配置docker

因为本次要部署的K8S版本为1.10.1版本，版本较低，所以docker版本不要安装太高的版本（亲测19.03版本不兼容）

yum install docker -y
systemctl start docker && systemctl enable docker

默认安装的是1.13版本（目前最新版本是1.13版本），不要安装docker-ce，docker-ce会安装19.03版本（截止目前的最新版本）查看docker cgroup

docker info

1.10.1版本的K8S使用的是cgroup driver，1.14之后的版本才推荐使用systemd，docker与K8S两者必须要保持一致，否则会报错

如果查看到的cgroup不为cgroupfs，需要修改

vim /usr/lib/systemd/system/docker.service

可以在添加一个加速

修改完成后重载配置，重启docker

systemctl daemon-reload && systemctl restart docker

查看docker版本

# docker --version
Docker version 1.13.1, build 0be3e21/1.13.1

④安装kubeadm，kubectl，kubelet

下载必要离线包，因为不下载的话，后面在初始化kubeadm时，会去自动拉去必要镜像，有些镜像需要翻墙才能下得下来

安装包连接🔗

链接：https://pan.baidu.com/s/1tPGxcqUkepbGnVV934bOpQ 提取码：r0cz 复制这段内容后打开百度网盘手机App，操作更方便哦

安装，此步骤6台机器全部都需要执行注：本文将所有K8S的文件放置在/root目录下,以下代码在使用时记得修改为实际路径

cd /root/kubernetes-1.10
tar -xvf kube-packages-1.10.1.tar
cd kube-packages-1.10.1
rpm -Uvh * --force --nodeps

在所有kubernetes节点上设置kubelet使用cgroupfs，与dockerd保持一致，否则kubelet会启动报错默认kubelet使用的cgroup-driver=systemd

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

建议：/etc/systemd/system/kubelet.service.d/10-kubeadm.conf文件中添加
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/k8sth/pause-amd64:3.0"

systemctl daemon-reload && systemctl restart kubelet && systemctl enable kubelet

导入镜像，只导入了必要镜像，后续镜像很多的话，可以考虑搭建harbor存放镜像，此步骤三天机器全部都需要执行

cd /root/kubernetes-1.10/
docker load -i k8s-images-1.10.tar.gz

命令补全

yum -y  install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

⑤初始化集群

node01、node02、node03添加集群初始配置文件（集群配置文件一样）

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://192.168.1.10:2379
  - https://192.168.1.20:2379
  - https://192.168.1.30:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.10.1
api:
  advertiseAddress: "192.168.1.10"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- node01
- 192.168.1.10
featureGates:
  CoreDNS: true
imageRepository: "registry.cn-beijing.aliyuncs.com/k8sct"

这个config

首先node01初始化集群

配置文件定义podnetwork是10.244.0.0/16

kubeadmin init –hlep可以看出，service默认网段是10.96.0.0/12

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf默认dns地址cluster-dns=10.96.0.10

kubeadm init --config config.yaml

初始化失败后处理办法

kubeadm reset
rm -rf $HOME/.kube
#或
rm -rf $HOME/.kube
rm -rf /etc/kubernetes/*.conf
rm -rf /etc/kubernetes/manifests/*.yaml
docker ps -a |awk '{print $1}' |xargs docker rm -f
systemctl  stop kubelet

初始化正常的结果如下

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.1.10:6443 --token b99a00.a144ef80536d4344 --discovery-token-ca-cert-hash sha256:7e234163db10f31e0fbb0c383410b81b8bd32f89fae1b947ab3f4ca75bd2f058

node01上面执行如下命令

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubeadm生成证书密码文件分发到node02和node03上面去（一会添加master节点）

scp -r /etc/kubernetes/pki  node02:/etc/kubernetes/
scp -r /etc/kubernetes/pki  node03:/etc/kubernetes/

部署flannel网络，只需要在node01执行就行

cd /root/kubernetes-1.10
kubectl apply -f kube-flannel.yml

查看状态kubernetes节点状态

[root@node01 kubernetes-1.10]# kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
node01    Ready     master    3m        v1.10.1
[root@node01 kubernetes-1.10]# kubectl get pods --all-namespaces 
NAMESPACE     NAME                             READY     STATUS    RESTARTS   AGE
kube-system   coredns-7997f8864c-85wds         1/1       Running   0          2m
kube-system   coredns-7997f8864c-9wjtx         1/1       Running   0          2m
kube-system   kube-apiserver-node01            1/1       Running   0          1m
kube-system   kube-controller-manager-node01   1/1       Running   0          2m
kube-system   kube-flannel-ds-ls2hp            1/1       Running   0          2m
kube-system   kube-proxy-77zkv                 1/1       Running   0          2m
kube-system   kube-scheduler-node01            1/1       Running   0          1m

将node02和node03节点同样执行

kubeadm init --config config.yaml

node02和node03输出的内容应该是与node01一致此时已经以master角色加入到集群了

node04、node05、node06执行命令加入集群以node角色加入

node01查看集群状态

[root@node01 ~]# kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
node01    Ready     master    10m       v1.10.1
node02    Ready     master    4m        v1.10.1
node03    Ready     master    4m        v1.10.1
node04    Ready     <none>    39s       v1.10.1
node05    Ready     <none>    27s       v1.10.1
node06    Ready     <none>    17s       v1.10.1

[root@node01 ~]# kubectl get pods --all-namespaces 
NAMESPACE     NAME                             READY     STATUS    RESTARTS   AGE
kube-system   coredns-7997f8864c-85wds         1/1       Running   0          13m
kube-system   coredns-7997f8864c-9wjtx         1/1       Running   0          13m
kube-system   kube-apiserver-node01            1/1       Running   0          12m
kube-system   kube-apiserver-node02            1/1       Running   0          7m
kube-system   kube-apiserver-node03            1/1       Running   0          7m
kube-system   kube-controller-manager-node01   1/1       Running   0          12m
kube-system   kube-controller-manager-node02   1/1       Running   0          7m
kube-system   kube-controller-manager-node03   1/1       Running   0          7m
kube-system   kube-flannel-ds-4dmg7            1/1       Running   0          7m
kube-system   kube-flannel-ds-8whpg            1/1       Running   1          3m
kube-system   kube-flannel-ds-g66s5            1/1       Running   0          3m
kube-system   kube-flannel-ds-j5dk6            1/1       Running   0          4m
kube-system   kube-flannel-ds-ls2hp            1/1       Running   0          12m
kube-system   kube-flannel-ds-s4vcz            1/1       Running   0          7m
kube-system   kube-proxy-4vm9g                 1/1       Running   0          4m
kube-system   kube-proxy-5mpng                 1/1       Running   0          7m
kube-system   kube-proxy-77zkv                 1/1       Running   0          13m
kube-system   kube-proxy-f67wb                 1/1       Running   0          7m
kube-system   kube-proxy-n4tlk                 1/1       Running   0          3m
kube-system   kube-proxy-q8sbm                 1/1       Running   0          3m
kube-system   kube-scheduler-node01            1/1       Running   0          12m
kube-system   kube-scheduler-node02            1/1       Running   0          7m
kube-system   kube-scheduler-node03            1/1       Running   0          7m

⑥Dashboard部署

直接使用离线包中的三个.yaml文件即可完成部署

cd /root/k8s/kubernetes-1.10
kubectl apply -f kubernetes-dashboard-http.yaml -f admin-role.yaml -f kubernetes-dashboard-admin.rbac.yaml

完成后查看主机是否起了31000端口，如果已启动，可通过ip:31000打开 K8S可视化页面

[root@node01 kubernetes-1.10]# netstat -tunlp | grep 31000
tcp6       0      0 :::31000                :::*                    LISTEN      56139/kube-proxy

⑦Rancher部署

docker安装rancher

docker run -d --name rancher --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher rancher/rancher:v2.2.4

界面访问直接访问IP

第一条授权经测试不执行也可以添加成功。如果集群有证书，直接第三条命令，执行：

[root@node01 kubernetes-1.10]# curl --insecure -sfL https://192.168.1.10/v3/import/5s2xmsfnbrj89thgf4m25pm9j4s6mcczsg7rszwsc95zm49m6ndtm5.yaml | kubectl apply -f -
namespace "cattle-system" created
serviceaccount "cattle" created
clusterrolebinding.rbac.authorization.k8s.io "cattle-admin-binding" created
secret "cattle-credentials-664b64d" created
clusterrole.rbac.authorization.k8s.io "cattle-admin" created
deployment.extensions "cattle-cluster-agent" created
daemonset.extensions "cattle-node-agent" created

查看rancher agent：

[root@node01 ~]# kubectl  get pod -n cattle-system
NAME                                    READY     STATUS    RESTARTS   AGE
cattle-cluster-agent-6559655864-p44qs   1/1       Running   0          1m
cattle-node-agent-9tnvs                 1/1       Running   0          39s
cattle-node-agent-c9x69                 1/1       Running   0          51s
cattle-node-agent-pt2r6                 1/1       Running   0          59s

回到浏览器发现集群已经倒入,集群能正常使用