二进制安装k8s集群(5)-安装flannel
在上一篇文章里我们主要介绍etcd集群的安装，这里我们主要介绍安装flannel。这里我们采用下载二进制binary制作linux systemd的方式安装，使用双向ssl的方式连接到etcd集群(因为我们的etcd集群开启了双向ssl)。安装完成之后我们要在etcd里创建k8s pod的通讯网络，启动flanneld生成docker绑定的环境变量，然后把这些环境变量设置到docker的启动参数里。最后由于我们使用的是flannel host-gw的网络方案，是属于underlay方式，需要host扮演路由的角色。那么还需要在每个host上开启网络转发的功能，同时也要accept pod cidr网络空间的流量。
创建flannel配置目录
mkdir -p /etc/flanneld
mkdir -p /var/lib/flanneld
chown -R root:root /etc/flanneld
chown -R root:root /var/lib/flanneld
下载并解压flannel：
注意这里在github下载flannel(flannel-v0.10.0-linux-amd64.tar.gz)
mkdir -p /opt/sw/flanneld/install
cd /opt/sw/flanneld/install/
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz
cp ./flanneld /usr/bin/
cp ./mk-docker-opts.sh /etc/flanneld创建flannel配置文件：
这里面会开启etcd的双向ssl认证，所以请提前制作好etcd client证书(可以参考以前文章中制作docker server cert的步骤)，并且copy到相应的配置目录里(这里是/etc/flanneld)
touch /etc/flanneld/flanneld.conf
chown -R root:root /etc/flanneld
cat > /etc/flanneld/flanneld.conf << EOF
FLANNEL_ETCD_ENDPOINTS="https://172.20.11.41:2379,https://172.20.11.42:2379,https://172.20.11.43:2379"
# etcd config key, this is the configuration key that flannel queries, for address range assignment
FLANNEL_ETCD_PREFIX="/cloudnetwork"
# Any additional options that you want to pass
FLANNEL_OPTIONS="-iface=enp0s3 \
-etcd-cafile=/etc/flanneld/ca.crt \
-etcd-certfile=/etc/flanneld/etcd-client.crt \
-etcd-keyfile=/etc/flanneld/etcd-client.key"
EOF
source /etc/flanneld/flanneld.conf
创建flannel systemd unit文件：
touch /usr/lib/systemd/system/flanneld.service
cat > /usr/lib/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld Server
After=network.target network-online.target etcd.service
Wants=network-online.target etcd.service
Before=docker.service
[Service]
Type=notify
WorkingDirectory=/var/lib/flanneld
EnvironmentFile=/etc/flanneld/flanneld.conf
User=root
ExecStartPre=source /etc/flanneld/flanneld.conf
ExecStart=/usr/bin/flanneld \
-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} \
-etcd-prefix=${FLANNEL_ETCD_PREFIX} \
${FLANNEL_OPTIONS}
ExecStartPost=/etc/flanneld/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
cat /usr/lib/systemd/system/flanneld.service
systemctl daemon-reload
在etcd存储里创建flanneld host-gw网络：
这里设置整个pod网络cidr为10.1.0.0/16，子网长度为24。请注意这里node的名字必须为config，flannel会按照配置文件的--etcd-prefix项下的config节点寻找配置信息。这里的--etcd-prefix项的值为cloudnetwork，所以配置节点的全路径为/cloudnetwork/config。
etcdctl --ca-file /etc/etcd/ca.crt --cert-file /etc/etcd/etcd-client.crt --key-file /etc/etcd/etcd-client.key set /cloudnetwork/config '{ "Network": "10.1.0.0/16", "SubnetLen": 24, "Backend": {"Type":"host-gw"}}'
etcdctl --ca-file /etc/etcd/ca.crt --cert-file /etc/etcd/etcd-client.crt --key-file /etc/etcd/etcd-client.key get /cloudnetwork/config
修改etcd，flanneld，docker的unit文件：
etcd，flanneld，docker三个系统服务有了依赖关系，flanneld依赖etcd，docker依赖flanneld。也需要在systemd unit文件里体现，这样在系统启动时也可以按照这个依赖关系顺序启动服务
vi /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target network-online.target
Before=flanneld.service
Wants=network-online.target
[Install]
WantedBy=multi-user.target
RequiredBy=flanneld.service
systemctl daemon-reload
systemctl enable etcd
vi /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld Server
After=network.target network-online.target etcd.service
Before=docker.service
Wants=network-online.target etcd.service
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
systemctl daemon-reload
systemctl enable flanneld
vi /usr/lib/systemd/system/docker.service
After=network-online.target firewalld.service containerd.service flanneld.service
Wants=network-online.target flanneld.service
Requires=docker.socket flanneld.service
ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock$DOCKER_NETWORK_OPTIONS
systemctl daemon-reload
systemctl enable docker
注意: 对于docker service，必须在启动参数里加入$DOCKER_NETWORK_OPTIONS，当前host上的flanneld会
给当前host上的docker生成网络绑定参数的环境变量，这个是flanneld网络的核心，所以一定要注意这个步骤。
在其它2台vm环境上重复上述步骤完成flannel安装，并设置开机启动flannel，start flannel服务检查状态
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld
检查flannel为docker生成的网络环境变量配置文件：
cat /run/flannel/docker
source /run/flannel/docker
echo ${DOCKER_NETWORK_OPTIONS}
检查docker网络地址的改变
ip addr|grep docker
开启网络转发
flannel host-gw的underlay方式需要host扮演路由的角色。那么就需要在每个host上开启网络转发的功能，还要accept pod cidr网络空间的流量。
创建网络转发脚本
mkdir -p /etc/customerscript/forward
touch /etc/customerscript/forward/enable-forward.sh
chown -R root:root /etc/customerscript
cat > /etc/customerscript/forward/enable-forward.sh << EOF
echo 1 > /proc/sys/net/ipv4/ip_forward &&
iptables -I FORWARD -d 10.1.0.0/16 -p tcp -j ACCEPT &&
iptables -I FORWARD -d 10.1.0.0/16 -p udp -j ACCEPT
EOF
chmod 754 /etc/customerscript/forward/enable-forward.sh
创建网络转发脚本的系统服务
touch /usr/lib/systemd/system/enable-forward.service
cat > /usr/lib/systemd/system/enable-forward.service << EOF
[Unit]
Description=Enable linux forward and also TCP/UDP package forward for k8s network
After=network.target network-online.target docker.service
Wants=network-online.target docker.service
[Service]
Type=notify
ExecStart=/usr/bin/sh /etc/customerscript/forward/enable-forward.sh
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
cat enable-forward.service
reload服务，设置开机启动，start服务脚本
systemctl daemon-reload
systemctl enable enable-forward.service
systemctl start enable-forward.service
目前先写到这里，下一篇文章里我们开始介绍私有仓库harbor repo的创建。