二进制包20分钟快速安装部署 Kubernetes v1.14.0 集群

耕耘实录

发布于 2019-07-04 17:27:13

5820

发布于 2019-07-04 17:27:13

二进制包20分钟快速安装部署 Kubernetes v1.14.0 集群

一环境

操作系统	Docker版本	Kubernetes版本	Etcd版本	Flannel版本
CentOS Linux release 7.6.1810	Docker version 18.09.4	v1.14.0	Version: 3.3.12	v0.11.0

二架构

主机名	IP	角色	部署应用
gysl-master	10.1.1.60	Msater	Docker/Kube-apiserver/kube-scheduler/kube-controller-manager/etcd
gysl-node1	10.1.1.61	Node	Docker/Kubelet/kube-proxy/flanneld/etcd
gysl-node2	10.1.1.62	Node	Docker/Kubelet/kube-proxy/flanneld/etcd
gysl-node3	10.1.1.63	Node	Docker/Kubelet/kube-proxy/flanneld/etcd

三安装过程

通过几个小时的努力，完成本次部署脚本的编写，安装脚本支持任意多个节点，主要通过三个脚本实现本次安装。

3.1 初始化脚本

#!/bin/bash
declare -A HostIP EtcdIP
HostIP=( [gysl-master]='10.1.1.60' [gysl-node1]='10.1.1.61' [gysl-node2]='10.1.1.62' [gysl-node3]='10.1.1.63' )
EtcdIP=( [etcd-master]='10.1.1.60' [etcd-01]='10.1.1.61' [etcd-02]='10.1.1.62' [etcd-03]='10.1.1.63' )
BinaryDir='/usr/local/bin'
KubeConf='/etc/kubernetes/conf.d'
KubeCA='/etc/kubernetes/ca.d'
EtcdConf='/etc/etcd/conf.d'
EtcdCA='/etc/etcd/ca.d'
FlanneldConf='/etc/flanneld'

mkdir -p {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}
for hostname in ${!HostIP[@]}
    do
        cat>>/etc/hosts<<EOF
${HostIP[${hostname}]} ${hostname}
EOF
    done
# Install the Docker engine. This needs to be executed on every machine.
curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo>&/dev/null
if [ $? -eq 0 ] ;
    then
        yum remove docker \
                      docker-client \
                      docker-client-latest \
                      docker-common \
                      docker-latest \
                      docker-latest-logrotate \
                      docker-logrotate \
                      docker-selinux \
                      docker-engine-selinux \
                      docker-engine>&/dev/null
        yum list docker-ce --showduplicates|grep "^doc"|sort -r
        yum -y install docker-ce-18.09.3-3.el7
        rm -f /etc/yum.repos.d/docker-ce.repo
        systemctl enable docker --now && systemctl status docker
    else
        echo "Install failed! Please try again! ";
        exit 110
fi
# Modify related kernel parameters. 
cat>/etc/sysctl.d/docker.conf<<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF 
sysctl -p /etc/sysctl.d/docker.conf>&/dev/null 
# Turn off and disable the firewalld.  
systemctl stop firewalld  
systemctl disable firewalld  
# Disable the SELinux.  
sed -i.bak 's/=enforcing/=disabled/' /etc/selinux/config  
# Disable the swap.  
sed -i.bak 's/^.*swap/#&/g' /etc/fstab
# Install EPEL/vim/git.  
yum -y install epel-release vim git tree
yum repolist
# Alias vim. 
cat>/etc/profile.d/vim.sh<<EOF
alias vi='vim'
EOF
source /etc/profile.d/vim.sh
echo "set nu">>/etc/vimrc
# Reboot the machine.  
reboot

需要每个节点都执行。

3.2 安装脚本

安装脚本较长，此处省略，安装日志供参考、拓展思路。此脚本在Master节点执行，安装过程无需连接外网，此过程也就是对之前文章的整合，详情请参阅：https://cloud.tencent.com/developer/article/1457333安装日志如下：

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rJdnEzx5GyWX9YCxq77ZMc+FCabCqA+3FwmS7LnF9qo Kubernetes
The key's randomart image is:
+---[RSA 1024]----+
|            .o. .|
|            .. +.|
|   . .      o + .|
|    + .. . . =   |
|   . + .S.= *    |
|    o ++o. B + o |
|    .++.=.* + o .|
|    .+ oo= + = . |
|    Eo+o  +.. o  |
+----[SHA256]-----+
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.62 (10.1.1.62)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.62's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@10.1.1.62'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.63 (10.1.1.63)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.63's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@10.1.1.63'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.61 (10.1.1.61)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.61's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@10.1.1.61'"
and check to make sure that only the key(s) you wanted were added.

etcd-v3.3.12-linux-amd64/
etcd-v3.3.12-linux-amd64/README.md
etcd-v3.3.12-linux-amd64/Documentation/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/interacting_v3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_concurrency_reference_v3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/limit.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/local_cluster.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_grpc_gateway.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/grpc_naming.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/experimental_apis.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_reference_v3.md
etcd-v3.3.12-linux-amd64/Documentation/integrations.md
etcd-v3.3.12-linux-amd64/Documentation/README.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-1-0-alpha-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-storage-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/README.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-watch-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-demo-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/rfc/
etcd-v3.3.12-linux-amd64/Documentation/rfc/v3api.md
etcd-v3.3.12-linux-amd64/Documentation/docs.md
etcd-v3.3.12-linux-amd64/Documentation/production-users.md
etcd-v3.3.12-linux-amd64/Documentation/metrics.md
etcd-v3.3.12-linux-amd64/Documentation/v2/
etcd-v3.3.12-linux-amd64/Documentation/v2/authentication.md
etcd-v3.3.12-linux-amd64/Documentation/v2/proxy.md
etcd-v3.3.12-linux-amd64/Documentation/v2/glossary.md
etcd-v3.3.12-linux-amd64/Documentation/v2/docker_guide.md
etcd-v3.3.12-linux-amd64/Documentation/v2/configuration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/README.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-1-0-alpha-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-storage-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/README.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-watch-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-demo-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/
etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/v3api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/libraries-and-tools.md
etcd-v3.3.12-linux-amd64/Documentation/v2/discovery_protocol.md
etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-configuration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_3.md
etcd-v3.3.12-linux-amd64/Documentation/v2/auth_api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/errorcode.md
etcd-v3.3.12-linux-amd64/Documentation/v2/admin_guide.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_2.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_1.md
etcd-v3.3.12-linux-amd64/Documentation/v2/clustering.md
etcd-v3.3.12-linux-amd64/Documentation/v2/other_apis.md
etcd-v3.3.12-linux-amd64/Documentation/v2/production-users.md
etcd-v3.3.12-linux-amd64/Documentation/v2/metrics.md
etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-reconf-design.md
etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules.yml
etcd-v3.3.12-linux-amd64/Documentation/v2/security.md
etcd-v3.3.12-linux-amd64/Documentation/v2/branch_management.md
etcd-v3.3.12-linux-amd64/Documentation/v2/internal-protocol-versioning.md
etcd-v3.3.12-linux-amd64/Documentation/v2/members_api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/
etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/freebsd.md
etcd-v3.3.12-linux-amd64/Documentation/v2/faq.md
etcd-v3.3.12-linux-amd64/Documentation/v2/backward_compatibility.md
etcd-v3.3.12-linux-amd64/Documentation/v2/04_to_2_snapshot_migration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules
etcd-v3.3.12-linux-amd64/Documentation/v2/api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/api_v3.md
etcd-v3.3.12-linux-amd64/Documentation/v2/reporting_bugs.md
etcd-v3.3.12-linux-amd64/Documentation/v2/tuning.md
etcd-v3.3.12-linux-amd64/Documentation/v2/dev/
etcd-v3.3.12-linux-amd64/Documentation/v2/dev/release.md
etcd-v3.3.12-linux-amd64/Documentation/branch_management.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/
etcd-v3.3.12-linux-amd64/Documentation/platforms/container-linux-systemd.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/freebsd.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/aws.md
etcd-v3.3.12-linux-amd64/Documentation/faq.md
etcd-v3.3.12-linux-amd64/Documentation/dl_build.md
etcd-v3.3.12-linux-amd64/Documentation/reporting_bugs.md
etcd-v3.3.12-linux-amd64/Documentation/tuning.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_4.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrading-etcd.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_2.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_1.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_0.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/logging.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/discovery_protocol.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/release.md
etcd-v3.3.12-linux-amd64/Documentation/learning/
etcd-v3.3.12-linux-amd64/Documentation/learning/auth_design.md
etcd-v3.3.12-linux-amd64/Documentation/learning/glossary.md
etcd-v3.3.12-linux-amd64/Documentation/learning/data_model.md
etcd-v3.3.12-linux-amd64/Documentation/learning/api_guarantees.md
etcd-v3.3.12-linux-amd64/Documentation/learning/why.md
etcd-v3.3.12-linux-amd64/Documentation/learning/api.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/
etcd-v3.3.12-linux-amd64/Documentation/op-guide/authentication.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/versioning.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/hardware.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/grafana.json
etcd-v3.3.12-linux-amd64/Documentation/op-guide/monitoring.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/configuration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/v2-migration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/maintenance.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-configuration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/recovery.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/clustering.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules
etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-reconf-design.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules.yml
etcd-v3.3.12-linux-amd64/Documentation/op-guide/security.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/performance.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd-sample-grafana.png
etcd-v3.3.12-linux-amd64/Documentation/op-guide/grpc_proxy.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/gateway.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/container.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/supported-platform.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/failures.md
etcd-v3.3.12-linux-amd64/Documentation/demo.md
etcd-v3.3.12-linux-amd64/README-etcdctl.md
etcd-v3.3.12-linux-amd64/etcdctl
etcd-v3.3.12-linux-amd64/READMEv2-etcdctl.md
etcd-v3.3.12-linux-amd64/etcd
flanneld
mk-docker-opts.sh
README.md
kubernetes/
kubernetes/server/
kubernetes/server/bin/
kubernetes/server/bin/kube-controller-manager.docker_tag
kubernetes/server/bin/kube-apiserver.tar
kubernetes/server/bin/kube-proxy
kubernetes/server/bin/cloud-controller-manager.docker_tag
kubernetes/server/bin/mounter
kubernetes/server/bin/kube-proxy.docker_tag
kubernetes/server/bin/kubelet
kubernetes/server/bin/kube-scheduler.docker_tag
kubernetes/server/bin/kube-controller-manager.tar
kubernetes/server/bin/kubectl
kubernetes/server/bin/kube-apiserver
kubernetes/server/bin/kube-apiserver.docker_tag
kubernetes/server/bin/kube-controller-manager
kubernetes/server/bin/kube-proxy.tar
kubernetes/server/bin/cloud-controller-manager
kubernetes/server/bin/kube-scheduler.tar
kubernetes/server/bin/apiextensions-apiserver
kubernetes/server/bin/kubeadm
kubernetes/server/bin/hyperkube
kubernetes/server/bin/kube-scheduler
kubernetes/server/bin/cloud-controller-manager.tar
kubernetes/addons/
kubernetes/kubernetes-src.tar.gz
kubernetes/LICENSES
2019/03/31 20:34:23 [INFO] generating a new CA key and certificate from CSR
2019/03/31 20:34:23 [INFO] generate received request
2019/03/31 20:34:23 [INFO] received CSR
2019/03/31 20:34:23 [INFO] generating key: rsa-2048
2019/03/31 20:34:23 [INFO] encoded CSR
2019/03/31 20:34:23 [INFO] signed certificate with serial number 316253512009054883826466120107550244311105093255
2019/03/31 20:34:23 [INFO] generate received request
2019/03/31 20:34:23 [INFO] received CSR
2019/03/31 20:34:23 [INFO] generating key: rsa-2048
2019/03/31 20:34:23 [INFO] encoded CSR
2019/03/31 20:34:23 [INFO] signed certificate with serial number 288189004496636237074496723049170901716100041831
2019/03/31 20:34:23 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
/etc/etcd/ca.d
├── ca-config.json
├── ca.csr
├── ca-csr.json
├── ca-key.pem
├── ca.pem
├── server.csr
├── server-csr.json
├── server-key.pem
└── server.pem

0 directories, 9 files
ca-key.pem                                                                                                                                                 100% 1675    27.1KB/s   00:00    
ca.pem                                                                                                                                                     100% 1265     2.2MB/s   00:00    
server-key.pem                                                                                                                                             100% 1679   639.6KB/s   00:00    
server.pem                                                                                                                                                 100% 1346     1.9MB/s   00:00    
etcd                                                                                                                                                       100%   18MB  13.1MB/s   00:01    
etcdctl                                                                                                                                                    100%   15MB  41.5MB/s   00:00    
etcd.service                                                                                                                                               100%  994     1.0MB/s   00:00    
etcd.conf                                                                                                                                                  100%  520   527.8KB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
ca-key.pem                                                                                                                                                 100% 1675    92.3KB/s   00:00    
ca.pem                                                                                                                                                     100% 1265     1.7MB/s   00:00    
server-key.pem                                                                                                                                             100% 1679   328.8KB/s   00:00    
server.pem                                                                                                                                                 100% 1346     1.6MB/s   00:00    
etcd                                                                                                                                                       100%   18MB  40.7MB/s   00:00    
etcdctl                                                                                                                                                    100%   15MB  46.0MB/s   00:00    
etcd.service                                                                                                                                               100%  994     1.0MB/s   00:00    
etcd.conf                                                                                                                                                  100%  520   838.6KB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
ca-key.pem                                                                                                                                                 100% 1675   106.9KB/s   00:00    
ca.pem                                                                                                                                                     100% 1265     1.0MB/s   00:00    
server-key.pem                                                                                                                                             100% 1679     1.3MB/s   00:00    
server.pem                                                                                                                                                 100% 1346     1.5MB/s   00:00    
etcd                                                                                                                                                       100%   18MB  31.4MB/s   00:00    
etcdctl                                                                                                                                                    100%   15MB  37.7MB/s   00:00    
etcd.service                                                                                                                                               100%  994   916.5KB/s   00:00    
etcd.conf                                                                                                                                                  100%  520   487.5KB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:37:32 CST; 28ms ago
 Main PID: 7373 (etcd)
    Tasks: 7
   Memory: 9.1M
   CGroup: /system.slice/etcd.service
           └─7373 /usr/local/bin/etcd --name=etcd-01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.61:2380 --listen-client-urls=https://10.1.1.61:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.61:2379 --initial-advertise-peer-urls=https://10.1.1.61:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem

3月 31 20:37:32 gysl-node1 etcd[7373]: 1c3555118a39401e initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
3月 31 20:37:32 gysl-node1 etcd[7373]: raft.node: 1c3555118a39401e elected leader 63ac3c747757aa28 at term 138
3月 31 20:37:32 gysl-node1 etcd[7373]: published {Name:etcd-01 ClientURLs:[https://10.1.1.61:2379]} to cluster 575c8b9e68fd927d
3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
3月 31 20:37:32 gysl-node1 etcd[7373]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
3月 31 20:37:32 gysl-node1 systemd[1]: Started Etcd Server.
3月 31 20:37:32 gysl-node1 etcd[7373]: serving client requests on 10.1.1.61:2379
3月 31 20:37:32 gysl-node1 etcd[7373]: set the initial cluster version to 3.0
3月 31 20:37:32 gysl-node1 etcd[7373]: enabled capabilities for version 3.0
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:37:34 CST; 124ms ago
 Main PID: 7551 (etcd)
    Tasks: 7
   Memory: 10.3M
   CGroup: /system.slice/etcd.service
           └─7551 /usr/local/bin/etcd --name=etcd-master --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.60:2380 --listen-client-urls=https://10.1.1.60:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.60:2379 --initial-advertise-peer-urls=https://10.1.1.60:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem

3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 63ac3c747757aa28 (stream Message reader)
3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 1c3555118a39401e (stream Message reader)
3月 31 20:37:34 gysl-master etcd[7551]: published {Name:etcd-master ClientURLs:[https://10.1.1.60:2379]} to cluster 575c8b9e68fd927d
3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
3月 31 20:37:34 gysl-master etcd[7551]: serving client requests on 10.1.1.60:2379
3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
3月 31 20:37:34 gysl-master etcd[7551]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
3月 31 20:37:34 gysl-master etcd[7551]: 78df1ab24a6f1302 initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 3 active peer(s)
3月 31 20:37:34 gysl-master systemd[1]: Started Etcd Server.
3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 76bcb3b85e42210d (stream Message reader)
Please wait a moment!
2019/03/31 20:38:34 [INFO] generating a new CA key and certificate from CSR
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:34 [INFO] encoded CSR
2019/03/31 20:38:34 [INFO] signed certificate with serial number 284879897535931954074635242912207100624264127544
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:34 [INFO] encoded CSR
2019/03/31 20:38:34 [INFO] signed certificate with serial number 163588537762519336822862885460408698694735647771
2019/03/31 20:38:34 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:35 [INFO] encoded CSR
2019/03/31 20:38:35 [INFO] signed certificate with serial number 269430846139878968754015022650791204259891937310
2019/03/31 20:38:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
/etc/kubernetes/ca.d
├── ca-config.json
├── ca.csr
├── ca-csr.json
├── ca-key.pem
├── ca.pem
├── kube-proxy.csr
├── kube-proxy-csr.json
├── kube-proxy-key.pem
├── kube-proxy.pem
├── server.csr
├── server-csr.json
├── server-key.pem
└── server.pem

0 directories, 13 files
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:38:35 CST; 41ms ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 7628 (kube-apiserver)
    Tasks: 1
   Memory: 14.0M
   CGroup: /system.slice/kube-apiserver.service
           └─7628 /usr/local/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.1.1.60:2379,https://10.1.1.61:2379,https://10.1.1.62:2379,https://10.1.1.63:2379 --bind-address=10.1.1.60 --secure-port=6443 --advertise-address=10.1.1.60 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/conf.d/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/etc/kubernetes/ca.d/server.pem --tls-private-key-file=/etc/kubernetes/ca.d/server-key.pem --client-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-key-file=/etc/kubernetes/ca.d/ca-key.pem --etcd-cafile=/etc/etcd/ca.d/ca.pem --etcd-certfile=/etc/etcd/ca.d/server.pem --etcd-keyfile=/etc/etcd/ca.d/server-key.pem

3月 31 20:38:35 gysl-master systemd[1]: Started Kubernetes API Server.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:38:36 CST; 20s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 7673 (kube-scheduler)
    Tasks: 7
   Memory: 47.5M
   CGroup: /system.slice/kube-scheduler.service
           └─7673 /usr/local/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect

3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.299502    7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.399931    7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.500642    7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.601146    7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.604604    7673 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705500    7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705529    7673 controller_utils.go:1034] Caches are synced for scheduler controller
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705631    7673 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-scheduler...
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.737674    7673 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.838862    7673 shared_informer.go:123] caches populated
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:38:56 CST; 20s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 7725 (kube-controller)
    Tasks: 6
   Memory: 132.3M
   CGroup: /system.slice/kube-controller-manager.service
           └─7725 /usr/local/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ca.d/ca.pem --cluster-signing-key-file=/etc/kubernetes/ca.d/ca-key.pem --root-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-private-key-file=/etc/kubernetes/ca.d/ca-key.pem

3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.915581    7725 request.go:530] Throttling request took 1.356935667s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1?timeout=32s
3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.965276    7725 request.go:530] Throttling request took 1.406608026s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1beta1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.015978    7725 request.go:530] Throttling request took 1.457255375s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.065993    7725 request.go:530] Throttling request took 1.507246887s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1beta1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067050    7725 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[/v1, Resource=configmaps:{} /v1, Resource=endpoints:{} /v1, Resource=events:{} /v1, Resource=limitranges:{} /v1, Resource=persistentvolumeclaims:{} /v1, Resource=pods:{} /v1, Resource=podtemplates:{} /v1, Resource=replicationcontrollers:{} /v1, Resource=resourcequotas:{} /v1, Resource=secrets:{} /v1, Resource=serviceaccounts:{} /v1, Resource=services:{} apps/v1, Resource=controllerrevisions:{} apps/v1, Resource=daemonsets:{} apps/v1, Resource=deployments:{} apps/v1, Resource=replicasets:{} apps/v1, Resource=statefulsets:{} autoscaling/v1, Resource=horizontalpodautoscalers:{} batch/v1, Resource=jobs:{} batch/v1beta1, Resource=cronjobs:{} coordination.k8s.io/v1, Resource=leases:{} events.k8s.io/v1beta1, Resource=events:{} extensions/v1beta1, Resource=daemonsets:{} extensions/v1beta1, Resource=deployments:{} extensions/v1beta1, Resource=ingresses:{} extensions/v1beta1, Resource=networkpolicies:{} extensions/v1beta1, Resource=replicasets:{} networking.k8s.io/v1, Resource=networkpolicies:{} networking.k8s.io/v1beta1, Resource=ingresses:{} policy/v1beta1, Resource=poddisruptionbudgets:{} rbac.authorization.k8s.io/v1, Resource=rolebindings:{} rbac.authorization.k8s.io/v1, Resource=roles:{}]
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067168    7725 resource_quota_monitor.go:180] QuotaMonitor unable to use a shared informer for resource "extensions/v1beta1, Resource=networkpolicies": no informer found for extensions/v1beta1, Resource=networkpolicies
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067189    7725 resource_quota_monitor.go:243] quota synced monitors; added 0, kept 30, removed 0
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: E0331 20:39:00.067197    7725 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
3月 31 20:39:13 gysl-master kube-controller-manager[7725]: I0331 20:39:13.677245    7725 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
3月 31 20:39:14 gysl-master kube-controller-manager[7725]: I0331 20:39:14.215322    7725 pv_controller_base.go:407] resyncing PV controller
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
member 1c3555118a39401e is healthy: got healthy result from https://10.1.1.61:2379
member 63ac3c747757aa28 is healthy: got healthy result from https://10.1.1.63:2379
member 76bcb3b85e42210d is healthy: got healthy result from https://10.1.1.62:2379
member 78df1ab24a6f1302 is healthy: got healthy result from https://10.1.1.60:2379
cluster is healthy
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
kubelet                                                                                                                                                    100%  122MB  26.1MB/s   00:04    
kube-proxy                                                                                                                                                 100%   35MB  21.9MB/s   00:01    
flanneld                                                                                                                                                   100%   34MB  20.5MB/s   00:01    
mk-docker-opts.sh                                                                                                                                          100% 2139   916.8KB/s   00:00    
flanneld.conf                                                                                                                                              100%  247    55.8KB/s   00:00    
flanneld.service                                                                                                                                           100%  389    82.7KB/s   00:00    
kubelet.yaml                                                                                                                                               100%  263   319.4KB/s   00:00    
kubelet.conf                                                                                                                                               100%  365   326.0KB/s   00:00    
kube-proxy.conf                                                                                                                                            100%  158   184.0KB/s   00:00    
kubelet.service                                                                                                                                            100%  267   234.2KB/s   00:00    
kube-proxy.service                                                                                                                                         100%  234   130.5KB/s   00:00    
bootstrap.kubeconfig                                                                                                                                       100% 2163     1.5MB/s   00:00    
kube-proxy.kubeconfig                                                                                                                                      100% 6265     4.4MB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:27 CST; 430ms ago
  Process: 7536 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
 Main PID: 7508 (flanneld)
    Tasks: 7
   Memory: 6.7M
   CGroup: /system.slice/flanneld.service
           └─7508 /usr/local/bin/flanneld --ip-masq

3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.720837    7508 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.721919    7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.722994    7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.724549    7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.730116    7508 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737143    7508 main.go:429] Waiting for 22h59m59.914613166s to renew lease
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737262    7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.744276    7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
3月 31 20:39:27 gysl-node2 systemd[1]: Started Flanneld overlay address etcd agent.
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.766442    7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:28 CST; 10ms ago
     Docs: https://docs.docker.com
 Main PID: 7579 (dockerd)
    Tasks: 8
   Memory: 32.1M
   CGroup: /system.slice/docker.service
           └─7579 /usr/bin/dockerd --bip=172.17.24.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock

3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843896719+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843917442+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, CONNECTING" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843973658+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, READY" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.844332744+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848229255+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848828116+08:00" level=info msg="Loading containers: start."
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.081132437+08:00" level=info msg="Loading containers: done."
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167227705+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167281411+08:00" level=info msg="Daemon has completed initialization"
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.175538228+08:00" level=info msg="API listen on /var/run/docker.sock"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:28 CST; 61ms ago
 Main PID: 7727 (kubelet)
    Tasks: 1
   Memory: 2.1M
   CGroup: /system.slice/kubelet.service
           └─7727 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.62 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

3月 31 20:39:28 gysl-node2 systemd[1]: Started Kubernetes Kubelet.

● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:28 CST; 7ms ago
 Main PID: 7728 (systemd)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/kube-proxy.service
           └─7728 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
kubelet                                                                                                                                                    100%  122MB  27.7MB/s   00:04    
kube-proxy                                                                                                                                                 100%   35MB  13.8MB/s   00:02    
flanneld                                                                                                                                                   100%   34MB  33.6MB/s   00:01    
mk-docker-opts.sh                                                                                                                                          100% 2139     1.1MB/s   00:00    
flanneld.conf                                                                                                                                              100%  247   225.5KB/s   00:00    
flanneld.service                                                                                                                                           100%  389   357.8KB/s   00:00    
kubelet.yaml                                                                                                                                               100%  263   193.7KB/s   00:00    
kubelet.conf                                                                                                                                               100%  365   331.3KB/s   00:00    
kube-proxy.conf                                                                                                                                            100%  158   130.4KB/s   00:00    
kubelet.service                                                                                                                                            100%  267   295.5KB/s   00:00    
kube-proxy.service                                                                                                                                         100%  234   198.3KB/s   00:00    
bootstrap.kubeconfig                                                                                                                                       100% 2163     2.0MB/s   00:00    
kube-proxy.kubeconfig                                                                                                                                      100% 6265     3.7MB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:39 CST; 391ms ago
  Process: 7534 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
 Main PID: 7502 (flanneld)
    Tasks: 7
   Memory: 9.3M
   CGroup: /system.slice/flanneld.service
           └─7502 /usr/local/bin/flanneld --ip-masq

3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088309    7502 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088315    7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.091984    7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.095011    7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.098419    7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.099751    7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.103532    7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.106520    7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.113480    7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:39 gysl-node3 systemd[1]: Started Flanneld overlay address etcd agent.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:39 CST; 10ms ago
     Docs: https://docs.docker.com
 Main PID: 7573 (dockerd)
    Tasks: 8
   Memory: 31.9M
   CGroup: /system.slice/docker.service
           └─7573 /usr/bin/dockerd --bip=172.17.100.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock

3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230510356+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230556184+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, CONNECTING" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230711652+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, READY" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.231101930+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234478410+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234950238+08:00" level=info msg="Loading containers: start."
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.406988224+08:00" level=info msg="Loading containers: done."
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497837879+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497901197+08:00" level=info msg="Daemon has completed initialization"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.502801194+08:00" level=info msg="API listen on /var/run/docker.sock"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:39 CST; 58ms ago
 Main PID: 7721 (kubelet)
    Tasks: 1
   Memory: 4.2M
   CGroup: /system.slice/kubelet.service
           └─7721 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.63 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Kubelet.

● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:39 CST; 21ms ago
 Main PID: 7722 (systemd)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/kube-proxy.service
           └─7722 /usr/lib/systemd/systemd --switched-root --system --deserialize 22

3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Proxy.
kubelet                                                                                                                                                    100%  122MB  32.7MB/s   00:03    
kube-proxy                                                                                                                                                 100%   35MB  14.4MB/s   00:02    
flanneld                                                                                                                                                   100%   34MB  30.4MB/s   00:01    
mk-docker-opts.sh                                                                                                                                          100% 2139     3.5MB/s   00:00    
flanneld.conf                                                                                                                                              100%  247   227.9KB/s   00:00    
flanneld.service                                                                                                                                           100%  389   359.0KB/s   00:00    
kubelet.yaml                                                                                                                                               100%  263   197.9KB/s   00:00    
kubelet.conf                                                                                                                                               100%  365   517.1KB/s   00:00    
kube-proxy.conf                                                                                                                                            100%  158   244.5KB/s   00:00    
kubelet.service                                                                                                                                            100%  267   379.4KB/s   00:00    
kube-proxy.service                                                                                                                                         100%  234   324.5KB/s   00:00    
bootstrap.kubeconfig                                                                                                                                       100% 2163   429.6KB/s   00:00    
kube-proxy.kubeconfig                                                                                                                                      100% 6265     4.7MB/s   00:00    
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:49 CST; 319ms ago
  Process: 7580 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
 Main PID: 7550 (flanneld)
    Tasks: 7
   Memory: 6.7M
   CGroup: /system.slice/flanneld.service
           └─7550 /usr/local/bin/flanneld --ip-masq

3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.414921    7550 vxlan_network.go:60] watching for new subnet leases
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.415303    7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.416682    7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.418320    7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.438055    7550 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443066    7550 main.go:429] Waiting for 22h59m59.922013672s to renew lease
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443213    7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:49 gysl-node1 systemd[1]: Started Flanneld overlay address etcd agent.
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.459736    7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.469674    7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:49 CST; 9ms ago
     Docs: https://docs.docker.com
 Main PID: 7622 (dockerd)
    Tasks: 8
   Memory: 28.6M
   CGroup: /system.slice/docker.service
           └─7622 /usr/bin/dockerd --bip=172.17.96.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock

3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549105373+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549111902+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549148708+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, CONNECTING" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549210269+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, READY" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549578647+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.554893473+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.555866350+08:00" level=info msg="Loading containers: start."
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.695192119+08:00" level=info msg="Loading containers: done."
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729225641+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729282016+08:00" level=info msg="Daemon has completed initialization"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:50 CST; 50ms ago
 Main PID: 7770 (kubelet)
    Tasks: 1
   Memory: 2.1M
   CGroup: /system.slice/kubelet.service
           └─7770 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.61 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Kubelet.

● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2019-03-31 20:39:50 CST; 20ms ago
 Main PID: 7771 (systemd)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/kube-proxy.service
           └─7771 /usr/lib/systemd/systemd --switched-root --system --deserialize 22

3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Proxy.
[root@gysl-master ~]# kubectl get cs,nodes
NAME                                 STATUS    MESSAGE             ERROR
componentstatus/scheduler            Healthy   ok
componentstatus/controller-manager   Healthy   ok
componentstatus/etcd-0               Healthy   {"health":"true"}
componentstatus/etcd-2               Healthy   {"health":"true"}
componentstatus/etcd-1               Healthy   {"health":"true"}
componentstatus/etcd-3               Healthy   {"health":"true"}

NAME             STATUS   ROLES   AGE     VERSION
node/10.1.1.61   Ready    node    4m23s   v1.14.0
node/10.1.1.62   Ready    node    4m22s   v1.14.0
node/10.1.1.63   Ready    node    4m22s   v1.14.0

3.3 安装失败回滚脚本

#!/bin/bash
declare -A HostIP EtcdIP
HostIP=( [gysl-master]='10.1.1.60' [gysl-node1]='10.1.1.61' [gysl-node2]='10.1.1.62' [gysl-node3]='10.1.1.63' )
EtcdIP=( [etcd-master]='10.1.1.60' [etcd-01]='10.1.1.61' [etcd-02]='10.1.1.62' [etcd-03]='10.1.1.63' )
BinaryDir='/usr/local/bin'
KubeConf='/etc/kubernetes/conf.d'
KubeCA='/etc/kubernetes/ca.d'
EtcdConf='/etc/etcd/conf.d'
EtcdCA='/etc/etcd/ca.d'
FlanneldConf='/etc/flanneld'
for node_ip in ${HostIP[@]}
    do
        if [ "${node_ip}" == "${HostIP[gysl-master]}" ] ; then
            ps -ef|grep -e kube -e etcd -e flanneld|grep -v grep|awk '{print $2}'|xargs kill 
            rm -rf {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}
            rm -rf ${BinaryDir}/*
        else
            ssh root@${node_ip} "ps -ef|grep -e kube -e etcd -e flanneld|grep -v grep|awk '{print $2}'|xargs kill"
            ssh root@${node_ip} "rm -rf {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}"
            ssh root@${node_ip} "rm -rf ${BinaryDir}/* && reboot"
        fi
    done
reboot