国内环境Kubernetes v1.12.1的安装与配置

版权声明:本文为耕耘实录原创文章,各大自媒体平台同步更新。欢迎转载,转载请注明出处,谢谢

一、些重要概念

Cluster:计算、存储、网络资源的总和。Kubernetes的各种基于容器的应用都是运行在这些资源上的。

Master:Kubernetes的大脑,负责调度各种计算资源。Master可以是物理机或虚拟机,多个Master可以同时运行,并实现HA。Master节点上运行的组件可以参见本文架构图。

Node:负责运行容器的应用,由Master管理,可以是物理机或虚拟机。

Pod:Kubernetes的最小工作单元,也就是说Kubernetes管理的是Pod而不是容器。每个Pod包括一个或多个容器。Pod中的容器会被作为一个整体被Master调度到另一个Node上。

Controller:Kubernetes通常不会直接创建Pod,而是通过Controller来管理Pod的。Controller中定义了容器中的一些部署特性。

Service:外界访问一组特定的Pod方式,有自己的IP和端口,Service为Pod提供了负载均衡。

Namespace:Namespace可以将一个物理的Cluster逻辑上划分为多个虚拟Cluster,每个Cluster就是一个Namespace,不同的Namespace里的资源完全是隔离的。创建资源时,如果不指定,将会被放到default这个默认的Namespace中。

二、实战环境

  1. 各节点操作系统版本。
[root@k8s-m ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core)
[root@k8s-m ~]# hostnamectl
   Static hostname: k8s-m
         Icon name: computer-vm
           Chassis: vm
        Machine ID: a9fff7a4819c41e1944be91c46dc15aa
           Boot ID: 03b55afaf51647c498d5b1df7ec7d331
    Virtualization: microsoft
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

所有主机版本相同,均为最小化安装版。 2. Docker版本。

[root@k8s-m ~]# docker -v
Docker version 18.06.0-ce, build 0ffa825
  1. Kubernetes版本。
[root@k8s-m ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
  1. 各主机的主机名及ip配置。 本次实战中一共用到三台主机,一台用于Master的部署,领导两台分别为node1和node2。主机名和IP的对应关系如下:
k8s-m 172.31.3.11
k8s-n1 172.31.3.12
k8s-n2 172.31.3.13

三、架构图

四、操作步骤

4.1每台主机都必须执行的操作

  1. 关闭并禁用防火墙。
[root@k8s-m ~]# systemctl stop firewalld
[root@k8s-m ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  1. 关闭SeLinux。
[root@k8s-m ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config
  1. 禁用swap。
# swapoff -a
[root@k8s-m ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.7G        115M        1.4G        8.9M        219M        1.4G
Swap:            0B          

如果要永久禁止swap挂载,可以修改/etc/fstab,将与swap有关的配置注释,重启系统即可。

[root@k8s-m ~]# reboot
[root@k8s-m ~]# getenforce
Disabled
  1. 修改/etc/hosts文件。
[root@k8s-m ~]# echo "172.31.3.11 k8s-m
172.31.3.12 k8s-n1
172.31.3.13 k8s-n2">>/etc/hosts
  1. 配置Docker的yum安装源,并安装docker-ce-18.06.0。
[root@k8s-m ~]# echo '[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg'>/etc/yum.repos.d/docker-ce.repo
[root@k8s-m ~]# yum list docker-ce --showduplicates|grep "^doc"|sort -r
docker-ce.x86_64            18.06.1.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.0.ce-3.el7                    docker-ce-stable
...

yum安装指定的Docker版本,安装完成后启动docker,并设置开机自动启动。

[root@k8s-m ~]# yum -y install docker-ce-18.06.0.ce-3.el7
[root@k8s-m ~]# systemctl start docker
[root@k8s-m ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
  1. 开启ipv4的转发在CentOS7.5版本上依然是必须的。如果使用到ipv6可能还需要启用ipv6的转发,操作类似。之前文章有介绍过。
[root@k8s-m ~]# echo "net.ipv4.ip_forward = 1">>/etc/sysctl.conf
[root@k8s-m ~]# sysctl -p
  1. 启用bridge-nf-call-iptables和bridge-nf-call-ip6tables。

docker info 命令看一下,出现以下警告,按照如下操作流程进行解决:

WARNING: bridge-nf-call-iptables is disabled

WARNING: bridge-nf-call-ip6tables is disabled

[root@k8s-m ~]# echo 'net.bridge.bridge-nf-call-iptables = 1'>>/etc/sysctl.conf
[root@k8s-m ~]# echo 'net.bridge.bridge-nf-call-ip6tables = 1'>>/etc/sysctl.conf
[root@k8s-m ~]# sysctl -p
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
  1. 配置kubernetes阿里云yum镜像。
[root@k8s-m ~]# echo '[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg'>/etc/yum.repos.d/kubernetes.repo
  1. 在每个节点上安装kubeadm,kubelet,kubectl。
[root@k8s-m ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
已安装:
  kubeadm.x86_64 0:1.12.1-0                  kubectl.x86_64 0:1.12.1-0                  kubelet.x86_64 0:1.12.1-0
作为依赖被安装:
  cri-tools.x86_64 0:1.12.0-0              kubernetes-cni.x86_64 0:0.6.0-0              socat.x86_64 0:1.7.3.2-2.el7
[root@k8s-m ~]# systemctl enable kubelet && systemctl start kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

以上步骤需要在所有机器上同步配置执行。

4.2在master k8s-m上执行

[root@k8s-m ~] kubeadm init --kubernetes-version=1.12.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.3.11
  1. 记录错误:
[root@k8s-m ~]# kubeadm init --kubernetes-version=1.12.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.3.11
[init] using Kubernetes version: v1.12.1
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.12.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.12.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.12.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.12.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
  1. 根据报错信息,在国内网站站上找到相关的镜像。
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.12.1
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.12.1
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.12.1
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.12.1
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.2.24
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
[root@k8s-m ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.2.2
[root@k8s-m ~]# docker images
REPOSITORY                                                                          TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64                v1.12.1             61afff57f010        12 days ago         96.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64            v1.12.1             dcb029b5e3ad        12 days ago         194MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64   v1.12.1             aa2dd57c7329        12 days ago         164MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64            v1.12.1             d773ad20fd80        12 days ago         58.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64                      3.2.24              3cab8e1b9802        3 weeks ago         220MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                         1.2.2               367cdc8433a4        7 weeks ago         39.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                           3.1                 da86e6ba6ca1        10 months ago       742kB
  1. 把这些images重新tag一下。
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.2.24 k8s.gcr.io/etcd:3.2.24
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.12.1 k8s.gcr.io/kube-scheduler:v1.12.1
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.12.1 k8s.gcr.io/kube-controller-manager:v1.12.1
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.12.1 k8s.gcr.io/kube-apiserver:v1.12.1
[root@k8s-m ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.12.1 k8s.gcr.io/kube-proxy:v1.12.1
  1. 编辑以下配置文件/var/lib/kubelet/kubeadm-flags.env,编辑后内容如下(在本次安装过程中,安装完成之后自动创建该文件,无需执行本步骤。在之前版本中这一步或许必不可少,具体情况请具体分析。本步骤主要是核实cgroup-driver=cgroupfs):
[root@k8s-m ~]# echo 'KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs  --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni'>/var/lib/kubelet/kubeadm-flags.env
[root@k8s-m ~]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
  1. 再次初始化Master。

知道配置之后,我又重新部署了一台机器,以下是操作记录。如果我们在部署过程中出现错误,我们可以使用kubeadm reset命令来重置。

[root@k8s-m ~]# kubeadm init --kubernetes-version=1.12.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.3.11
[init] using Kubernetes version: v1.12.1
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [k8s-m localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-m localhost] and IPs [172.31.3.11 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-m kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.3.11]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 27.015529 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node k8s-m as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node k8s-m as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-m" as an annotation
[bootstraptoken] using token: jlidhg.sczmdc5wnvgi5pir
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.31.3.11:6443 --token jlidhg.sczmdc5wnvgi5pir --discovery-token-ca-cert-hash sha256:74f5e3a3b1163fdafe9a634ff5ca81ea314a6895af18e03a5b482635b4ca93e0

[root@k8s-m ~]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni
[root@k8s-m ~]#  mkdir -p $HOME/.kube
[root@k8s-m ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-m ~]# chown $(id -u):$(id -g) $HOME/.kube/config

初始化Master节点之后,查看各节点的时间是否一致,如果不一致则需要再次同步个节点的时间。时间同步可以看之前NTP相关的文章。保存安装日志的以下部分以备接下来在各node执行:

kubeadm join 172.31.3.11:6443 --token jlidhg.sczmdc5wnvgi5pir --discovery-token-ca-cert-hash sha256:74f5e3a3b1163fdafe9a634ff5ca81ea314a6895af18e03a5b482635b4ca93e0
  1. 安装flanel。
[root@k8s-m ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

4.3在node k8s-n1和k8s-n2分别执行以下命令(可以从升级日志直接复制)。

[root@k8s-n1 ~]# kubeadm join 172.31.3.11:6443 --token jlidhg.sczmdc5wnvgi5pir --discovery-token-ca-cert-hash sha256:74f5e3a3b1163fdafe9a634ff5ca81ea314a6895af18e03a5b482635b4ca93e0
[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

[discovery] Trying to connect to API Server "172.31.3.11:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.31.3.11:6443"
[discovery] Requesting info from "https://172.31.3.11:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.3.11:6443"
[discovery] Successfully established connection with API Server "172.31.3.11:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-n1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.
[root@k8s-n2 ~]# kubeadm join 172.31.3.11:6443 --token jlidhg.sczmdc5wnvgi5pir --discovery-token-ca-cert-hash sha256:74f5e3a3b1163fdafe9a634ff5ca81ea314a6895af18e03a5b482635b4ca93e0
[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

[discovery] Trying to connect to API Server "172.31.3.11:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.31.3.11:6443"
[discovery] Requesting info from "https://172.31.3.11:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.3.11:6443"
[discovery] Successfully established connection with API Server "172.31.3.11:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-n2" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

查看以下各node的状态。

[root@k8s-m ~]# kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
k8s-m    Ready      master   67m   v1.12.1
k8s-n1   NotReady   <none>   30m   v1.12.1
k8s-n2   NotReady   <none>   21m   v1.12.1

再查看一下各pod的情况。

[root@k8s-m ~]# kubectl get pod --all-namespaces
NAMESPACE     NAME                            READY   STATUS              RESTARTS   AGE
kube-system   coredns-576cbf47c7-h9fst        1/1     Running             0          23m
kube-system   coredns-576cbf47c7-wnzz5        1/1     Running             0          23m
kube-system   etcd-k8s-m                      1/1     Running             0          68m
kube-system   kube-apiserver-k8s-m            1/1     Running             0          69m
kube-system   kube-controller-manager-k8s-m   1/1     Running             0          68m
kube-system   kube-flannel-ds-amd64-2d4qc     1/1     Running             0          49m
kube-system   kube-flannel-ds-amd64-2fc6q     0/1     Init:0/1            0          32m
kube-system   kube-flannel-ds-amd64-7725c     0/1     Init:0/1            0          23m
kube-system   kube-proxy-jw7dw                1/1     Running             0          69m
kube-system   kube-proxy-qtgl8                0/1     ContainerCreating   0          32m
kube-system   kube-proxy-r4gx9                0/1     ContainerCreating   0          23m
kube-system   kube-scheduler-k8s-m            1/1     Running             0          69m

五、清除警告和错误

1.在k8s-n1和k8s-n2上执行把该节点注册到集群时,出现以下警告。

[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

在最小化安装的CentOS7.5系统中没有包含改内核模块。对一般的实验和简单的使用没有影响。在后期会有专门的文章对此问题进行解决。

2.执行以下命令kubectl get pod --all-namespaces后,我们看到有的pod的状态并不是running,使用命令kubectl describe pod查看详情后,在屏幕末端显示警告和错误如下:

[root@k8s-m ~]# kubectl describe pod kube-flannel-ds-amd64-2fc6q --namespace=kube-system
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               33m                   default-scheduler  Successfully assigned kube-system/kube-flannel-ds-amd64-2fc6q to k8s-n1
  Warning  FailedCreatePodSandBox  3m34s (x61 over 33m)  kubelet, k8s-n1    Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

按照错误提示,我们推断产生这样的报错是因为有的镜像未被pull,本文环境不具备科学上网条件,无法从Google直接pull镜像,所以我们必须按照错误提示去pull相关镜像,并重新tag。分析得知,除Master之外,每个node还缺一些镜像,按照如下方法操作即可解决问题。

[root@k8s-n1 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
[root@k8s-n1 ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
[root@k8s-n1 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.12.1
[root@k8s-n1 ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.12.1 k8s.gcr.io/kube-proxy:v1.12.1
[root@k8s-n1 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/kubernetes_containers/flannel:v0.10.0-amd64
[root@k8s-n1 ~]# docker tag registry.cn-hangzhou.aliyuncs.com/kubernetes_containers/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64

现在,我们再看一下:

[root@k8s-m ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   coredns-576cbf47c7-h9fst        1/1     Running   0          105m
kube-system   coredns-576cbf47c7-wnzz5        1/1     Running   0          105m
kube-system   etcd-k8s-m                      1/1     Running   0          151m
kube-system   kube-apiserver-k8s-m            1/1     Running   0          152m
kube-system   kube-controller-manager-k8s-m   1/1     Running   0          151m
kube-system   kube-flannel-ds-amd64-2d4qc     1/1     Running   0          132m
kube-system   kube-flannel-ds-amd64-2fc6q     1/1     Running   0          115m
kube-system   kube-flannel-ds-amd64-7725c     1/1     Running   0          106m
kube-system   kube-proxy-jw7dw                1/1     Running   0          152m
kube-system   kube-proxy-qtgl8                1/1     Running   0          115m
kube-system   kube-proxy-r4gx9                1/1     Running   0          106m
kube-system   kube-scheduler-k8s-m            1/1     Running   0          151m
[root@k8s-m ~]# kubectl get nodes
NAME     STATUS   ROLES    AGE    VERSION
k8s-m    Ready    master   153m   v1.12.1
k8s-n1   Ready    <none>   116m   v1.12.1
k8s-n2   Ready    <none>   107m   v1.12.1
[root@k8s-m ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

六、总结

  1. 本文中介绍的kubernetes部署方式不建议使用到生产环境,可以用于实验环境和测试环境。
  2. kubernetes的部署过程中,网络的访问是一个问题,如果解决了这个问题,kubernetes的部署会显得格外简单。
  3. 当试错成本可控时,在上生产环境之前请尽情地去试错,并记录、分析各种错误产生的原因。
  4. 养成记录安装日志的习惯,不要随便忽略每一个警告和错误。

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏大魏分享(微信公众号:david-share)

从一张图看Devops全流程

一、持续交付工具链全图 ? 上图源自网络。上图很清晰地列出了CD几个阶段使用的工具。 CD的工具链很长,但并不是每个模块所有工具都那么流行;换言之,我们在每个模...

1.6K9
来自专栏运维

Openstack G版本 Ubuntu13.04三节点实验记录

本篇文档参考了官网文档(http://docs.openstack.org/),github(https://github.com/mseknibilel/Op...

1402
来自专栏Linux运维学习之路

Docker集群编排工具之Kubernetes(K8s)介绍、安装及使用

K8s基础原理 k8s中文社区:https://www.kubernetes.org.cn/ 简介 Kubernetes与较早的集群管理系统Mesos和YARN...

1.8K5
来自专栏Jerry的SAP技术分享

Kubernetes Helm入门指南

什么是Helm?这可不是暗黑破坏神里装备的名称:头盔,而是Kubernetes的一个包管理工具,用来简化Kubernetes应用的部署和管理。我们Helm和Ku...

2030
来自专栏bboysoul

中间人攻击工具(Xerosploit)

Xerosploit是一个渗透测试工具包,它的目的是实现中间人攻击。它附带着各种有效的攻击模块,并且还允许执行拒绝服务攻击和端口扫描

1123
来自专栏康怀帅的专栏

CoreOS 容器 Rkt 简单介绍

由于 Docker 已经成为事实上的容器老大,这里暂且将 rkt 内容放入 docker 文件夹。哈哈 官方网站:https://coreos.com/rkt/...

4857
来自专栏从零学习云计算

kubernetes学习记录(10)——建立Heapster+Influxdb+Grafana集群性能监控平台

采用的是Heapster+Influxdb+Grafana建立集群性能监控平台。 据说Heapster需要与Kubernetes Master进行安全连接,所以...

4870
来自专栏数据和云

故障分析:RAC环境下ORA-12545连接错误

崔华,网名 dbsnake Oracle ACE Director,ACOUG 核心专家 分享一则由于SCAN IP导致的连接故障 11gR2 RAC两节点的环...

2984
来自专栏张善友的专栏

服务器同时存在1.1和2.0程序注意事项

Asp.net和asp.net2.0同时部署在服务器上:发生如下错误 Server Application Unavailable The web appli...

1968
来自专栏大魏分享(微信公众号:david-share)

隆重介绍!CI/CD手下的开源界六大金刚

Jenkins 2 image based on Red Hat Enterprise Linux的镜像

4833

扫码关注云+社区

领取腾讯云代金券