利用Multipass搭建k8s

是小张啊喂

发布于 2022-08-18 17:31:11

9040

发布于 2022-08-18 17:31:11

文章被收录于专栏：软件软件

Multipass如何使用可以参考之前的文章multipass使用初体验

搭建部署过程

关闭交换空间

$ swapoff -a

查看交换空间

$ free -h

避免开机启动交换空间

$ vi /etc/fstab
注释掉swap开头的行

关闭防火墙

$ ufw disable

安装docker

自行安装

安装Kubernetes必备工具

kubeadm
kubelet
kubectl

安装系统工具

$ apt-get update && apt-get install -y apt-transport-https

安装CPG证书

$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg  | apt-key add -

写入软件源

$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

安装

$ apt-get update && apt-get install -y kubeadm kubelet kubectl

E: The repository 'https://download.docker.com/linux/ubuntu focal Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

清空 /etc/apt/sources.list.d/docker.list

重新安装，没有遇到直接跳过

$ apt-get update && apt-get install -y kubeadm kubelet kubectl

到此以上三台步骤需要全部执行,很重要

设置时区

$ dpkg-reconfigure tzdata

安装 ntpdate

$ apt-get install ntpdate

设置系统时间与网格同步

$ ntpdate edu.ntp.org.cn

参考网址：http://www.ntp.org.cn/pool

将系统时间写入硬件时间
$ hwclock --systohc

修改cloud.cfg

$ vi /etc/cloud/cloud.cfg
# 修改成 true
preserve_hostname: true

multipass 已经给做好了

$ hostnamectl set-hostname master

$ cat >> /etc/hosts<<EOF
192.168.141.110 master
EOF

安装kubernetes

$ cd /usr/local/
$ mkdir kubernetes
$ cd kubernetes
$ mkdir cluster
$ cd cluster

$ kubeadm config print init-defaults --kubeconfig ClusterConfiguration > kubeadm.yml

$ vim  kubeadm.yml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: 1.23.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

修改主机ip
advertiseAddress
  
修改镜像地址
k8s.gcr.io 为Google镜像地址
imageRepository：registry.aliyuncs.com/google_containers
注意：
kubernetesVersion 版本
#podSubnet ：”10.224.0.0/16“

查看需要下载哪些镜像

$ kubeadm config images list --config kubeadm.yml
registry.aliyuscn.com/google_containers/kube-apiserver:v1.23.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.23.0
registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/etcd:3.5.1-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6

拉取镜像

$ kubeadm config images pull --config kubeadm.yml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.6
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.1-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6
$ docker images

安装主节点

$ kubeadm init --config=kubeadm.yml --upload-certs | tee kubeadm-init.log

将其添加到/etc/docker/daemon.json,将dockercgroup驱动程序设置为systemd：

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

此时的daemon.json

{
    "registry-mirrors": ["https://c8lfvm3n.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]
}

重新加载daemon.json，重启docker和kubelet

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
$ sudo systemctl restart kubelet

[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[ERROR Port-10250]: Port 10250 is in use
[ERROR Port-2379]: Port 2379 is in use
[ERROR Port-2380]: Port 2380 is in use
[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty

遇到端口占用或者是加入失败，执行这个
$ kubeadm reset

$ kubeadm init --config=kubeadm.yml --upload-certs | tee kubeadm-init.log

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.19.47.43:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:840bc14e93004e5cbffd30bfc4a7ea7a3be3c78eb02f1a24fb75d482da0d6dc5
        
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config    

$ kubectl get nodes
NAME   STATUS     ROLES                  AGE     VERSION
node   NotReady   control-plane,master   9m32s   v1.23.3

在其余node节点上执行上面的加入集群的命令

$ kubeadm join 172.19.47.43:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:840bc14e93004e5cbffd30bfc4a7ea7a3be3c78eb02f1a24fb75d482da0d6dc5
        
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0128 11:22:03.577678   86042 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster

$ kubectl get nodes
kubectl get nodes
NAME    STATUS     ROLES                  AGE     VERSION
node    NotReady   control-plane,master   31m     v1.23.3
node1   NotReady   <none>                 8m28s   v1.23.3
node2   NotReady   <none>                 34s     v1.23.3

到此节点已经全部安装完毕，现在需要准备kubernetes网络

安装网络插件 Calico

什么是Calico？

Calico 是一套开源的网络和网络安全方案，用于容器、虚拟机、宿主机之前的网络连接，可以用在kubernetes、OpenShift、DockerEE、OpenStrack等PaaS或IaaS平台上。 Calico 还提供网络安全规则的动态实施，使用Calico的简单策略语言，您可以实现对容器，虚拟机工作负载和裸机主机断点之间通信的细粒度控制

官方文档：http://docs.projectcalico.org

https://projectcalico.docs.tigera.io/getting-started/kubernetes/quickstart

$ kubectl apply -f http://docs.projectcalico.org/v3.8/manifests/calico.yaml
$ kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
$ kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
$ kubectl get pods --all -namespances
$ watch kubectl get pods -n calico-system

Kubernetes 安装第一个容器

$ kubectl get cs

主节点信息
$ kubectl cluster-info

检查Nodes状态
$ kubectl get nodes
NAME    STATUS   ROLES                  AGE    VERSION
node    Ready    control-plane,master   165m   v1.23.3
node1   Ready    <none>                 142m   v1.23.3
node2   Ready    <none>                 134m   v1.23.3

运行第一个容器实例
# 创建一个pod
$ kubectl run nginx --image=nginx --  --port=80
pod/nginx created

$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          2m13s

# 部署一个nginx
$ kubectl create deployment nginx --image=nginx
pod/nginx created

# 获取部署额容器
$ kubectl get deployment
NAME                     READY   STATUS    RESTARTS   AGE
nginx-85b98978db-nsp5b   1/1     Running   0          56s

# 负载均衡器发布服务
$ kubectl expose deployment nginx --port=80 --type=LoadBalancer
service/nginx exposed

# 获取服务
$ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        3h5m
nginx        LoadBalancer   10.96.168.113   <pending>     80:30948/TCP   13s

# 获取服务详情
$  kubectl describe service nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.168.113
IPs:                      10.96.168.113
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30948/TCP
Endpoints:                192.168.104.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

访问 http://172.19.46.74:30948/ 验证是否成功

# 删除部署部署
$ kubectl delete deployment nginx
deployment.apps "nginx" deleted

@ 删除部署的服务
$ kubectl delete service nginx
service "nginx" deleted

通过资源配置运行容器

$ kubectl create -f
$ kubectl capply

$ vim ngxin.yml

# API 版本号
apiVersion: apps/v1
# 类型 如：Pod/ReplicationController/Deployment/Service/Ingress
kind: Deployment
# 元数据
metadata:
  # Kind 的名称
  name: nginx-app
spec:
  # 部署的实例数量
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template: 
    metadata:  
      labels:
        # 容器标签名称，发布 Service 时，selector 需要和这里对应
        app: nginx
    spec:
      # 配置容器，数组类型，说明可以配置多个容器
      containers:
      # 容器名称
      - name: nginx
        # 容器镜像
        image: nginx:1.17
        # 只有镜像不存在时才会进行镜像拉取
        imagePullPolicy: IfNotPresent
        # 暴露端口
        ports:
        # Pod 端口
        - containerPort: 80
        
$ kubectl create -f nginx.yml
deployment.apps/nginx-app created

extensions/v1beta1 API从kubernetes1.6之后以后不再支持，转而使用apps/v1

https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

spec.selector.matchLabels值和spec.template.metadata.lables值完全匹配

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
nginx-app-5757b68bb6-jqrsg   1/1     Running   0          3m30s
nginx-app-5757b68bb6-w8xvj   1/1     Running   0          3m30s

$ kubectl get deployment
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
nginx-app   2/2     2            2           3m45s

$ kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   8h

没有service

imagePullPolicy拉取策略

Always：不管镜像是否存在都会进行一次拉取
Never：不管镜像是否存在都不会进行拉取
IfNotPresent：只有镜像不存在时才会进行镜像拉取

注意

默认为IfNotPresent，但:latest标签的镜像默认为Always
拉取镜像时Docker会进行校验，如果镜像中的MD5码没有变，则不会拉取镜像
生产环境中应尽量避免使用:latest标签，而开发环境中可以借助:latest标签自动拉取最新的镜像

编辑nginx.yml

# API 版本号
apiVersion: apps/v1
# 类型 如：Pod/ReplicationController/Deployment/Service/Ingress
kind: Deployment
# 元数据
metadata:
  # Kind 的名称
  name: nginx-app
spec:
  # 部署的实例数量
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template: 
    metadata:  
      labels:
        # 容器标签名称，发布 Service 时，selector 需要和这里对应
        app: nginx
    spec:
      # 配置容器，数组类型，说明可以配置多个容器
      containers:
      # 容器名称
      - name: nginx
        # 容器镜像
        image: nginx:1.17
        # 只有镜像不存在时才会进行镜像拉取
        imagePullPolicy: IfNotPresent
        # 暴露端口
        ports:
        # Pod 端口
        - containerPort: 80
---
# API 版本号
apiVersion: v1
# 类型 如：Pod/ReplicationController/Deployment/Service/Ingress
kind: Service
# 元数据
metadata:
  # Kind 的名称
  name: nginx-http
spec:
  # 暴露端口
  ports:
    # Service 暴露的端口
    - port: 80
      # Pod 上的端口这里是将 Service 暴露的端口转发到 Pod 端口上
      targetPort: 80
  type: LoadBalancer
  # 标签选择器
  selector:
    # 需要和上面部署的 Deployment 标签名对应
    name: nginx
    

$  kubectl delete -f nginx.yml 
deployment.apps "nginx-app" deleted

$ kubectl create -f nginx.yml
deployment.apps/nginx-app created
service/nginx-http created

$ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        9h
nginx-http   LoadBalancer   10.97.228.234   <pending>     80:31641/TCP   114s

$ kubectl delete -f nginx.yml 
deployment.apps "nginx-app" deleted
service "nginx-http" deleted