IT苦工指南 | Kubernetes v1.8.x全手动安装

最近,有部分用户飘了……

觉得Rainbond提供的既简洁、又易用、而且生产就绪的Kubernets体验不过瘾……

想要挑战一下Kubernetes全手动部署……

并在凌晨一点拨通了客服小哥的电话……

因此本着不重复造轮子并且关爱客服小哥身心健康的主张,我们搬来了Kairen的精彩教程——

开始

Kubernetes官方提供了多种安装方式Picking the right solution,本文将以全手动安装方式来部署Kubernetes v1.8.x版本,学习和了解Kubernetes的构建流程。

版本明细:

* Kubernetes v1.8.6

* CNI v0.6.0

* Etcd v3.2.9

* Calico v2.6.2

* Docker v17.10.0-ce

准备

系统:ubuntu 16.xcentos 7.x

节点:

* 172.16.35.12 / master1 / 1 CPU / 2G

* 172.16.35.10 / node1 / 1 CPU / 2G

* 172.16.35.11 / node2 / 1 CPU / 2G

master为主要控制节点和部署节点,node为应用运行节点

所有操作均为root

安装前需确认以下事项:

* 确认所有节点之间网络互通,master1 SSH登入其他节点为passwordless

* 确认防火墙和SELinux已关闭,如centos:

$ systemctl stop firewalld && systemctl disable firewalld

$ setenforce 0

$ vim /etc/selinux/config

SELINUX=disabled

* 所有节点需要设置/etc/host解析到所有主机

...

172.16.35.10 node1

172.16.35.11 node2

172.16.35.12 master1

* 所有节点都需要安装docke

$ curl -fsSL "https://get.docker.com/" | sh 

注意:centos安装docker完成后需要执行:

$ systemctl enable docker && systemctl start docker 

编辑/lib/systemd/system/docker.service,在ExecStart=..加入:

ExecStartPost=/sbin/iptables -A FORWARD -s 0.0.0.0/0 -j ACCEPT 

完成后重启docker服务:

$ systemctl daemon-reload && systemctl restart docker 

* 所有节点都需要设定/etc/sysctl.d/k8s.conf系统参数

$ cat <<EOF > /etc/sysctl.d/k8s.conf

net.ipv4.ip\_forward = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF



$ sysctl -p /etc/sysctl.d/k8s.conf

* 在master1安装CFSSL工具,用来建立TLS certificates

$ export CFSSL\_URL="https://pkg.cfssl.org/R1.2"

$ wget "${CFSSL\_URL}/cfssl\_linux-amd64" -O /usr/local/bin/cfssl

$ wget "${CFSSL\_URL}/cfssljson\_linux-amd64" -O /usr/local/bin/cfssljson

$ chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

Etcd

安装Kubernetes之前,我们需要完成一些必要的系统配置,高可用共享配置和服务发现存储Etcd便是其中的重要一环,节点会从Etcd中获取所需数据。

建立集群CA和certificates

这里需要生成client和server各组件certificate,代替kubernetes admin user生成client证书。

首先在master1建立/etc/etcd/ssl目录,而后进入目录进行以下操作:

$ mkdir -p /etc/etcd/ssl && cd /etc/etcd/ssl

$ export PKI\_URL="https://kairen.github.io/files/manual-v1.8/pki"

下载ca-config.jsonetcd-ca-csr.json

$ wget "${PKI\_URL}/ca-config.json" "${PKI\_URL}/etcd-ca-csr.json"

$ cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca

$ ls etcd-ca\*.pem

etcd-ca-key.pem  etcd-ca.pem

下载etcd-csr.json并生成Etcd certificate证书:

$ wget "${PKI\_URL}/etcd-csr.json"

$ cfssl gencert \

  -ca=etcd-ca.pem \

  -ca-key=etcd-ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  etcd-csr.json | cfssljson -bare etcd



$ ls etcd\*.pem

etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem

如果节点IP不同,需要修改etcd-csr.json的hosts

完成后删除不必要的文件:

$ rm -rf \*.json 

并确认/etc/etcd/ssl包含:

$ ls /etc/etcd/ssl

etcd-ca.csr  etcd-ca-key.pem  etcd-ca.pem  etcd.csr  etcd-key.pem  etcd.pem

Etcd的安装和设置

首先在master1节点下载Etcd,解压到/opt安装:

$ export ETCD\_URL="https://github.com/coreos/etcd/releases/download"

$ cd && wget -qO- --show-progress "${ETCD\_URL}/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz" | tar -zx

$ mv etcd-v3.2.9-linux-amd64/etcd\* /usr/local/bin/ && rm -rf etcd-v3.2.9-linux-amd64

完成后新建Etcd Group和User,并设定Etcd目录:

$ groupadd etcd && useradd -c "Etcd user" -g etcd -s /sbin/nologin -r etcd

下载etcd相关配置,我们将来管理Etcd:

$ export ETCD\_CONF\_URL="https://kairen.github.io/files/manual-v1.8/master"

$ wget "${ETCD\_CONF\_URL}/etcd.conf" -O /etc/etcd/etcd.conf

$ wget "${ETCD\_CONF\_URL}/etcd.service" -O /lib/systemd/system/etcd.service

如果没用本文准备部分的IP,请用自己的IP代替172.16.35.12

建立var 存放数据,然后启动Etcd服务:

$ mkdir -p /var/lib/etcd && chown etcd:etcd -R /var/lib/etcd /etc/etcd

$ systemctl enable etcd.service && systemctl start etcd.service 

通过以下命令验证:

$ export CA="/etc/etcd/ssl"

$ ETCDCTL\_API=3 etcdctl \

    --cacert=${CA}/etcd-ca.pem \

    --cert=${CA}/etcd.pem \

    --key=${CA}/etcd-key.pem \

    --endpoints="https://172.16.35.12:2379" \

    endpoint health

# output

https://172.16.35.12:2379 is healthy: successfully committed proposal: took = 641.36µs

Kubernetes Maste

Master是Kubernetes的大总管,通过apiserverController manager以及Scheduler管理所有节点。

本部分将下载Kubernetes并安装到master1节点上,然后生成相关TLS certificates和CA,供集群组件使用。

下载Kubernetes组件

# Download Kubernetes

$ export KUBE\_URL="https://storage.googleapis.com/kubernetes-release/release/v1.8.6/bin/linux/amd64"

$ wget "${KUBE\_URL}/kubelet" -O /usr/local/bin/kubelet

$ wget "${KUBE\_URL}/kubectl" -O /usr/local/bin/kubectl

$ chmod +x /usr/local/bin/kubelet /usr/local/bin/kubectl



# Download CNI

$ mkdir -p /opt/cni/bin && cd /opt/cni/bin

$ export CNI\_URL="https://github.com/containernetworking/plugins/releases/download"

$ wget -qO- --show-progress "${CNI\_URL}/v0.6.0/cni-plugins-amd64-v0.6.0.tgz" | tar -zx

建立集群CA和certificates

与Etcd部分原理一样,操作也大相径庭,首先在master1建立pki目录,并进入目录执行:

$ mkdir -p /etc/kubernetes/pki && cd /etc/kubernetes/pki

$ export PKI\_URL="https://kairen.github.io/files/manual-v1.8/pki"

$ export KUBE\_APISERVER="https://172.16.35.12:6443"

下载ca-config.jsonetcd-ca-csr.json

$ wget "${PKI\_URL}/ca-config.json" "${PKI\_URL}/ca-csr.json"

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

$ ls ca\*.pem

ca-key.pem  ca.pem
API server certificate

下载apiserver-csr.json,生成kube-apiserver certificate证书:

$ wget "${PKI\_URL}/apiserver-csr.json"

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -hostname=10.96.0.1,172.16.35.12,127.0.0.1,kubernetes.default \

  -profile=kubernetes \

  apiserver-csr.json | cfssljson -bare apiserve



$ ls apiserver\*.pem

apiserver-key.pem  apiserver.pem

如果节点IP不同,需要修改-hostname

Front proxy certificate

下载front-proxy-ca-csr.json,生成Front proxy CA,Front proxy主要用在API aggregator上:

$ wget "${PKI\_URL}/front-proxy-ca-csr.json"

$ cfssl gencert \

  -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca



$ ls front-proxy-ca\*.pem

front-proxy-ca-key.pem  front-proxy-ca.pem

下载front-proxy-client-csr.json,生成front-proxy-client证书:

$ wget "${PKI\_URL}/front-proxy-client-csr.json"

$ cfssl gencert \

  -ca=front-proxy-ca.pem \

  -ca-key=front-proxy-ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  front-proxy-client-csr.json | cfssljson -bare front-proxy-client



$ ls front-proxy-client\*.pem

front-proxy-client-key.pem  front-proxy-client.pem
Bootstrap Token

手工方式生成CA非常麻烦,只适合少量机器,每次签证时都需要绑定Node IP,随着机器增加会带来很多的不便,因此这里使用TLS Bootstrapping的方式来进行授权,由apiserver自动为符合条件的Node发送证书授权加入集群。

做法是在kubelet启动时,向kuber-apiserver传送TLS Bootstrapping请求,而kube-apiserver验证kubelet请求的token是否与设定的一样,如果一样则自动生成Kuberlet证书和密钥。具体作法可以参考TLS bootstrapping

首先生成BOOTSTRAP\_TOKEN,并建立bootstrap.conf的kubeconfig:

$ export BOOTSTRAP\_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')

$ cat <<EOF > /etc/kubernetes/token.csv

${BOOTSTRAP\_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"

EOF



# bootstrap set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server=${KUBE\_APISERVER} \

    --kubeconfig=../bootstrap.conf



# bootstrap set-credentials

$ kubectl config set-credentials kubelet-bootstrap \

    --token=${BOOTSTRAP\_TOKEN} \

    --kubeconfig=../bootstrap.conf



# bootstrap set-context

$ kubectl config set-context default \

    --cluster=kubernetes \

    --user=kubelet-bootstrap \

   --kubeconfig=../bootstrap.conf



# bootstrap set default context

$ kubectl config use-context default --kubeconfig=../bootstrap.conf

如果想用CA的方式来认证,可以参考Kubelet certificate

Admin certificate

下载admin-csr.json,并生成admin certificate证书:

$ wget "${PKI\_URL}/admin-csr.json"

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  admin-csr.json | cfssljson -bare admin



$ ls admin\*.pem

admin-key.pem  admin.pem

然后执行一下命令生成名为admin.conf的kubeconfig:

# admin set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server=${KUBE\_APISERVER} \

    --kubeconfig=../admin.conf



# admin set-credentials

$ kubectl config set-credentials kubernetes-admin \

    --client-certificate=admin.pem \

    --client-key=admin-key.pem \

    --embed-certs=true \

    --kubeconfig=../admin.conf



# admin set-context

$ kubectl config set-context kubernetes-admin@kubernetes \

    --cluster=kubernetes \

    --user=kubernetes-admin \

    --kubeconfig=../admin.conf



# admin set default context

$ kubectl config use-context kubernetes-admin@kubernetes \

    --kubeconfig=../admin.conf
Controller manager certificate

下载manager-csr.json,并生成kube-controller-manager certificate证书:

$ wget "${PKI\_URL}/manager-csr.json"

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  manager-csr.json | cfssljson -bare controller-manage



$ ls controller-manager\*.pem

如果节点IP不同,需要修改manager-csr.json的hosts

然后执行命令生成名为controller-manager.conf的kubeconfig:

# controller-manager set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server=${KUBE\_APISERVER} \

    --kubeconfig=../controller-manager.conf



# controller-manager set-credentials

$ kubectl config set-credentials system:kube-controller-manager \

    --client-certificate=controller-manager.pem \

    --client-key=controller-manager-key.pem \

    --embed-certs=true \

    --kubeconfig=../controller-manager.conf



# controller-manager set-context

$ kubectl config set-context system:kube-controller-manager@kubernetes \

    --cluster=kubernetes \

    --user=system:kube-controller-manager \

    --kubeconfig=../controller-manager.conf



# controller-manager set default context

$ kubectl config use-context system:kube-controller-manager@kubernetes \

    --kubeconfig=../controller-manager.conf
Scheduler certificate

下载scheduler-csr.json,生成kube-scheduler certificate证书:

$ wget "${PKI\_URL}/scheduler-csr.json"

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  scheduler-csr.json | cfssljson -bare schedule



$ ls scheduler\*.pem

scheduler-key.pem  scheduler.pem

如果节点IP不同,需要修改scheduler-csr.json的hosts

然后执行一下命令生成名为scheduler.conf的kubeconfig:

# scheduler set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server=${KUBE\_APISERVER} \

    --kubeconfig=../scheduler.conf



# scheduler set-credentials

$ kubectl config set-credentials system:kube-scheduler \

    --client-certificate=scheduler.pem \

    --client-key=scheduler-key.pem \

    --embed-certs=true \

    --kubeconfig=../scheduler.conf



# scheduler set-context

$ kubectl config set-context system:kube-scheduler@kubernetes \

    --cluster=kubernetes \

    --user=system:kube-scheduler \

    --kubeconfig=../scheduler.conf



# scheduler set default context

$ kubectl config use-context system:kube-scheduler@kubernetes \

    --kubeconfig=../scheduler.conf
Kubelet master certificate

下载kubelet-csr.json,并生成master node certificate证书:

$ wget "${PKI\_URL}/kubelet-csr.json"

$ sed -i 's/$NODE/master1/g' kubelet-csr.json

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -hostname=master1,172.16.35.12 \

  -profile=kubernetes \

  kubelet-csr.json | cfssljson -bare kubelet



$ ls kubelet\*.pem

kubelet-key.pem  kubelet.pem

$NODE需要随节点名称不同而改变

然后执行一下命令生成名为kubelet.conf的kubeconfig:

# kubelet set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server=${KUBE\_APISERVER} \

    --kubeconfig=../kubelet.conf



# kubelet set-credentials

$ kubectl config set-credentials system:node:master1 \

    --client-certificate=kubelet.pem \

    --client-key=kubelet-key.pem \

    --embed-certs=true \

    --kubeconfig=../kubelet.conf



# kubelet set-context

$ kubectl config set-context system:node:master1@kubernetes \

    --cluster=kubernetes \

    --user=system:node:master1 \

    --kubeconfig=../kubelet.conf



# kubelet set default context

$ kubectl config use-context system:node:master1@kubernetes \

    --kubeconfig=../kubelet.conf
Service account key

Service account不需要CA认证,也就不需要CA来做Service account key的检查,这里我们建立一组private和public的密钥供Service account key使用:

$ openssl genrsa -out sa.key 2048 $ openssl rsa -in sa.key -pubout -out sa.pub $ ls sa.\* sa.key sa.pub 

完成后删除不必要文件:

$ rm -rf \*.json \*.csr 

确认/etc/kubernetes/etc/kubernetes/pki包含以下文件:

$ ls /etc/kubernetes/

admin.conf  bootstrap.conf  controller-manager.conf  kubelet.conf  pki  scheduler.conf  token.csv



$ ls /etc/kubernetes/pki

admin-key.pem  apiserver-key.pem  ca-key.pem  controller-manager-key.pem  front-proxy-ca-key.pem  front-proxy-client-key.pem  kubelet-key.pem  sa.key  scheduler-key.pem

admin.pem      apiserver.pem      ca.pem      controller-manager.pem      front-proxy-ca.pem      front-proxy-client.pem      kubelet.pem      sa.pub  scheduler.pem

安装Kubernetes 核心元件

下载Kubernetes核心组件Yaml文件,这里我们利用Kubernetes Statics Pod来建立Master核心组件,因此下载所有Static Pod文件到etc/kubernetes/manifests目录“

$ export CORE\_URL="https://kairen.github.io/files/manual-v1.8/master"

$ mkdir -p /etc/kubernetes/manifests && cd /etc/kubernetes/manifests

$ for FILE in apiserver manager scheduler; do

    wget "${CORE\_URL}/${FILE}.yml.conf" -O ${FILE}.yml

  done

同样的,如果IP与本文IP准备不同的话,需要修改apiserver.ymlmanager.yml、`

scheduler.yml`

apiserver中的NodeRestriction请参考Using Node Authorization

生成一个用来加密Etcd的key

$ head -c 32 /dev/urandom | base64

SUpbL4juUYyvxj3/gonV5xVEx8j769/99TSAf8YT/sQ=

/etc/kubernetes/目录建立encryption.yml的加密YAML文件:

$ cat <<EOF > /etc/kubernetes/encryption.yml

kind: EncryptionConfig

apiVersion: v1

resources:

  - resources:

      - secrets

    providers:

      - aescbc:

          keys:

            - name: key1

              secret: SUpbL4juUYyvxj3/gonV5xVEx8j769/99TSAf8YT/sQ=

      - identity: {}

EOF

Etcd加密可参考Encrypting data at rest

/etc/kubernetes/目录建立audit-policy.yml的auditing policay YAML文件:

$ cat <<EOF > /etc/kubernetes/audit-policy.yml

apiVersion: audit.k8s.io/v1beta1

kind: Policy

rules:

- level: Metadata

EOF

audit policy请参考Audit

下载kubelet.service相关文件来管理kubelet:

$ export KUBELET\_URL="https://kairen.github.io/files/manual-v1.8/master"

$ mkdir -p /etc/systemd/system/kubelet.service.d

$ wget "${KUBELET\_URL}/kubelet.service" -O /lib/systemd/system/kubelet.service

$ wget "${KUBELET\_URL}/10-kubelet.conf" -O /etc/systemd/system/kubelet.service.d/10-kubelet.conf

cluster-dnscluster-domain有变动,需要修改10-kubelet.conf

最后建立var并启动kubelet服务:

$ mkdir -p /var/lib/kubelet /var/log/kubernetes

$ systemctl enable kubelet.service && systemctl start kubelet.service

完成后需要一段时间来下载镜像文件并启动组件:

$ watch netstat -ntlp

tcp        0      0 127.0.0.1:10248         0.0.0.0:\*               LISTEN      23012/kubelet

tcp        0      0 127.0.0.1:10251         0.0.0.0:\*               LISTEN      22305/kube-schedule

tcp        0      0 127.0.0.1:10252         0.0.0.0:\*               LISTEN      22529/kube-controll

tcp6       0      0 :::6443                 :::\*                    LISTEN      22956/kube-apiserve

看到上述信息即表明服务启动正常,如果出现问题可通过docker cli查看

完成后,复制admin kubeconfig并通过以下命令验证:

$ cp /etc/kubernetes/admin.conf ~/.kube/config

$ kubectl get cs

NAME                 STATUS    MESSAGE              ERROR

etcd-0               Healthy   {"health": "true"}

scheduler            Healthy   ok

controller-manager   Healthy   ok



$ kubectl get node

NAME      STATUS     ROLES     AGE       VERSION

master1   NotReady   master    1m        v1.8.6



$ kubectl -n kube-system get po

NAME                              READY     STATUS    RESTARTS   AGE

kube-apiserver-master1            1/1       Running   0          4m

kube-controller-manager-master1   1/1       Running   0          4m

kube-scheduler-master1            1/1       Running   0          4m

确认服务能够执行logs等命令:

$ kubectl -n kube-system logs -f kube-scheduler-master1

Error from server (Forbidden): Forbidden (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-apiserver-master1)

出现403 Forbidden问题表明kube-apiserver user并没有nodes的权限

由于上述权限问题,我们需要建立一个apiserver-to-kubelet-rbac.yml来定义权限,以供我们执行logs、exec等命令:

$ cd /etc/kubernetes/

$ export URL="https://kairen.github.io/files/manual-v1.8/master"

$ wget "${URL}/apiserver-to-kubelet-rbac.yml.conf" -O apiserver-to-kubelet-rbac.yml

$ kubectl apply -f apiserver-to-kubelet-rbac.yml



# 測試 logs

$ kubectl -n kube-system logs -f kube-scheduler-master1

...

I1031 03:22:42.527697       1 leaderelection.go:184] successfully acquired lease kube-system/kube-schedule

Kubernetes Node

Node运行容器实例的节点,即工作节点。本部分我们会下载Kubernetes binary并建立node 的certificate来提供给节点注册认证用。Kubernetes使用Node Authorizer来提供Authorization mode,这种授权模式会替Kubelet生成API request。

开始前,我们先在master1将需要的ca和cert复制到Node节点上:

$ for NODE in node1 node2; do

    ssh ${NODE} "mkdir -p /etc/kubernetes/pki/"

    ssh ${NODE} "mkdir -p /etc/etcd/ssl"

    # Etcd ca and cert

    for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do

      scp /etc/etcd/ssl/${FILE} ${NODE}:/etc/etcd/ssl/${FILE}

    done

    # Kubernetes ca and cert

    for FILE in pki/ca.pem pki/ca-key.pem bootstrap.conf; do

      scp /etc/kubernetes/${FILE} ${NODE}:/etc/kubernetes/${FILE}

    done

  done

下载Kubernetes 元件

首先获取所有需要执行的文件:

# Download Kubernetes

$ export KUBE\_URL="https://storage.googleapis.com/kubernetes-release/release/v1.8.6/bin/linux/amd64"

$ wget "${KUBE\_URL}/kubelet" -O /usr/local/bin/kubelet

$ chmod +x /usr/local/bin/kubelet



# Download CNI

$ mkdir -p /opt/cni/bin && cd /opt/cni/bin

$ export CNI\_URL="https://github.com/containernetworking/plugins/releases/download"

$ wget -qO- --show-progress "${CNI\_URL}/v0.6.0/cni-plugins-amd64-v0.6.0.tgz" | tar -zx

设定Kubernetes node

下载Kubernetes相关文件,包括drop-in file、systemd service等:

$ export KUBELET\_URL="https://kairen.github.io/files/manual-v1.8/node"

$ mkdir -p /etc/systemd/system/kubelet.service.d

$ wget "${KUBELET\_URL}/kubelet.service" -O /lib/systemd/system/kubelet.service

$ wget "${KUBELET\_URL}/10-kubelet.conf" -O /etc/systemd/system/kubelet.service.d/10-kubelet.conf

如果cluster-dnscluster-domain有改变的话,需要修改10-kubelet.conf

然后在所有node建立var,并启动kubelet服务:

$ mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/kubernetes/manifests

$ systemctl enable kubelet.service && systemctl start kubelet.service

授权Kubernetes Node

重复完成所有节点后,在master1节点建立ClusterRoleBinding(因为我们采用的是TLS Bootstrapping):

$ kubectl create clusterrolebinding kubelet-bootstrap \

    --clusterrole=system:node-bootstrapper \

    --user=kubelet-bootstrap

在master进行验证,我们可以看到节点处于pending:

$ kubectl get cs

NAME                                                   AGE       REQUESTOR           CONDITION

node-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE   2s        kubelet-bootstrap   Pending

node-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE   2s        kubelet-bootstrap   Pending

通过kubectl,允许节点加入集群:

$ kubectl get csr | awk '/Pending/ {print $1}' | xargs kubectl certificate approve

certificatesigningrequest "node-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE" approved

certificatesigningrequest "node-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE" approved



$ kubectl get cs

NAME                                                   AGE       REQUESTOR           CONDITION

node-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE   30s       kubelet-bootstrap   Approved,Issued

node-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE   30s       kubelet-bootstrap   Approved,Issued



$ kubectl get no

NAME      STATUS     ROLES     AGE       VERSION

master1   NotReady   master    21m       v1.8.6

node1     NotReady   node      8s        v1.8.6

node2     NotReady   node      8s        v1.8.6

Kubernetes Core Addons 部署

完成以上所有步骤,我们还需要安装一些插件,比如kube-dns、kube-proxy等等。

Kube-proxy addon

Kube-proxy是实现Service的关键组件,kube-proxy会在每个节点上执行,然后监听API Server的Service和Endpoint变化,并根据变化执行iptables实现网络转发。

这里我们需要DaemonSet来执行,并需要生成一些certificate。

首先在master1下载kube-proxy-csr.json,并生成kube-proxy certificate证书:

$ export PKI\_URL="https://kairen.github.io/files/manual-v1.8/pki"

$ cd /etc/kubernetes/pki

$ wget "${PKI\_URL}/kube-proxy-csr.json" "${PKI\_URL}/ca-config.json"

$ cfssl gencert \

  -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes \

  kube-proxy-csr.json | cfssljson -bare kube-proxy



$ ls kube-proxy\*.pem

kube-proxy-key.pem  kube-proxy.pem

然后通过以下命令生成名为`kube-proxy.conf·的kubeconfig:

# kube-proxy set-cluste

$ kubectl config set-cluster kubernetes \

    --certificate-authority=ca.pem \

    --embed-certs=true \

    --server="https://172.16.35.12:6443" \

    --kubeconfig=../kube-proxy.conf



# kube-proxy set-credentials

$ kubectl config set-credentials system:kube-proxy \

    --client-key=kube-proxy-key.pem \

    --client-certificate=kube-proxy.pem \

    --embed-certs=true \

    --kubeconfig=../kube-proxy.conf



# kube-proxy set-context

$ kubectl config set-context system:kube-proxy@kubernetes \

    --cluster=kubernetes \

    --user=system:kube-proxy \

    --kubeconfig=../kube-proxy.conf



# kube-proxy set default context

$ kubectl config use-context system:kube-proxy@kubernetes \

    --kubeconfig=../kube-proxy.conf

删除不必要的文件:

$ rm -rf \*.json

确认/etc/kubernetes有以下文件:

$ ls /etc/kubernetes/

admin.conf        bootstrap.conf           encryption.yml  kube-proxy.conf  pki             token.csv

audit-policy.yml  controller-manager.conf  kubelet.conf    manifests        scheduler.conf

master1上将kube-proxy相关文件复制到Node节点上:

$ for NODE in node1 node2; do

    echo "--- $NODE ---"

    for FILE in pki/kube-proxy.pem pki/kube-proxy-key.pem kube-proxy.conf; do

      scp /etc/kubernetes/${FILE} ${NODE}:/etc/kubernetes/${FILE}

    done

  done

完成后,在master1通过kubectl建立kube-proxy daemon:

$ export ADDON\_URL="https://kairen.github.io/files/manual-v1.8/addon"

$ mkdir -p /etc/kubernetes/addons && cd /etc/kubernetes/addons

$ wget "${ADDON\_URL}/kube-proxy.yml.conf" -O kube-proxy.yml

$ kubectl apply -f kube-proxy.yml

$ kubectl -n kube-system get po -l k8s-app=kube-proxy

NAME               READY     STATUS    RESTARTS   AGE

kube-proxy-bpp7q   1/1       Running   0          47s

kube-proxy-cztvh   1/1       Running   0          47s

kube-proxy-q7mm4   1/1       Running   0          47s

Kube-dns addon

Kube DNS是Kubernetes集群内部Pod之间通信的重要插件,允许Pod通过Domain Name链接Service,主要由Kube DNS与Sky DNS组合而成,通过Kube DNS监听Service与Endpoint变化,来提供给Sky DNS信息以更新解析位址。

安装只需要在master1通过kubectl建立kube-dns deployment即可:

$ export ADDON\_URL="https://kairen.github.io/files/manual-v1.8/addon"

$ wget "${ADDON\_URL}/kube-dns.yml.conf" -O kube-dns.yml

$ kubectl apply -f kube-dns.yml

$ kubectl -n kube-system get po -l k8s-app=kube-dns

NAME                        READY     STATUS    RESTARTS   AGE

kube-dns-6cb549f55f-h4zr5   0/3       Pending   0          40s

Calico Network 安装与设定

Calico是一款纯3层协议(不需要Overlay 网路),已与各种云原生平台有良好的整合,在每个节点节点利用Linux Kernel实现高效的vRouter来负责数据转发,而当数据中心复杂度增加时,可以用BGP route reflector来达成。

首先在master1通过kubectl建立Calico policy controller:

$ export CALICO\_CONF\_URL="https://kairen.github.io/files/manual-v1.8/network"

$ wget "${CALICO\_CONF\_URL}/calico-controller.yml.conf" -O calico-controller.yml

$ kubectl apply -f calico-controller.yml

$ kubectl -n kube-system get po -l k8s-app=calico-policy

NAME                                        READY     STATUS    RESTARTS   AGE

calico-policy-controller-5ff8b4549d-tctmm   0/1       Pending   0          5s

如果节点IP不同,需要修改calico-controller.yml的ETCD_ENDPOINTS

在`master1·下载Calico CLI工具:

$ wget https://github.com/projectcalico/calicoctl/releases/download/v1.6.1/calicoctl

$ chmod +x calicoctl && mv calicoctl /usr/local/bin/

然后在所有节点下载Calico,并执行以下命令:

$ export CALICO\_URL="https://github.com/projectcalico/cni-plugin/releases/download/v1.11.0"

$ wget -N -P /opt/cni/bin ${CALICO\_URL}/calico

$ wget -N -P /opt/cni/bin ${CALICO\_URL}/calico-ipam

$ chmod +x /opt/cni/bin/calico /opt/cni/bin/calico-ipam

接着在所有节点下载CNI plugins以及calico-node.service:

$ mkdir -p /etc/cni/net.d

$ export CALICO\_CONF\_URL="https://kairen.github.io/files/manual-v1.8/network"

$ wget "${CALICO\_CONF\_URL}/10-calico.conf" -O /etc/cni/net.d/10-calico.conf

$ wget "${CALICO\_CONF\_URL}/calico-node.service" -O /lib/systemd/system/calico-node.service

如果节点IP不同,需要修改10-calico.conf的etcd_endpoints

如果部署机器是虚拟机,需要修改calico-node.service,并在IP_AUTODETECTION_METHOD (包含IP6)部分指定绑定的网卡,以免预设绑定到NAT网路上

之后在所有节点启动Calico-node:

$ systemctl enable calico-node.service && systemctl start calico-node.service

master1查看Calico nodes:

$ cat <<EOF > ~/calico-rc

export ETCD\_ENDPOINTS="https://172.16.35.12:2379"

export ETCD\_CA\_CERT\_FILE="/etc/etcd/ssl/etcd-ca.pem"

export ETCD\_CERT\_FILE="/etc/etcd/ssl/etcd.pem"

export ETCD\_KEY\_FILE="/etc/etcd/ssl/etcd-key.pem"

EOF



$ . ~/calico-rc

$ calicoctl get node -o wide

NAME      ASN       IPV4              IPV6

master1   (64512)   172.16.35.12/24

node1     (64512)   172.16.35.10/24

node2     (64512)   172.16.35.11/24

查看pending的pod是否已执行:

$ kubectl -n kube-system get po

NAME                                        READY     STATUS    RESTARTS   AGE

calico-policy-controller-5ff8b4549d-tctmm   1/1       Running   0          4m

kube-apiserver-master1                      1/1       Running   0          20m

kube-controller-manager-master1             1/1       Running   0          20m

kube-dns-6cb549f55f-h4zr5                   3/3       Running   0          5m

kube-proxy-fnrkb                            1/1       Running   0          6m

kube-proxy-l72bq                            1/1       Running   0          6m

kube-proxy-m6rfw                            1/1       Running   0          6m

kube-scheduler-master1                      1/1       Running   0          20m

省事的做法是用Standard Hosted方式安装。

Kubernetes Extra Addons 部署

本部分说明如何部署官方常用的addons,例如dashboard、heapster等。

Dashboard addon

Dashboard是Kubernetes官方开发的仪表板,让我们以可以i 通过web-based方式管理kubernetes集群。

master1通过kubectl建立kubernetes dashboard即可:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

$ kubectl -n kube-system get po,svc -l k8s-app=kubernetes-dashboard

NAME                                      READY     STATUS    RESTARTS   AGE

po/kubernetes-dashboard-747c4f7cf-md5m8   1/1       Running   0          56s



NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE

svc/kubernetes-dashboard   ClusterIP   10.98.120.209   <none>        443/TCP   56s

这边会额外建立一个名称为open-api的Cluster Role Binding,放拜年测试使用,一般情况下不开启(开启会存取所有API)。

$ cat <<EOF | kubectl create -f -

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: open-api

  namespace: ""

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

  - apiGroup: rbac.authorization.k8s.io

    kind: Use

    name: system:anonymous

EOF

管理者可以针对特定使用者来开放API存取权限,这里我们为了方便直接绑在cluster-admin cluster role。

1.7版本后的Dashboard不再提供所有权限,需要建立一个service account来绑定cluster-admin role:

$ kubectl -n kube-system create sa dashboard

$ kubectl create clusterrolebinding dashboard --clusterrole cluster-admin --serviceaccount=kube-system:dashboard

$ SECRET=$(kubectl -n kube-system get sa dashboard -o yaml | awk '/dashboard-token/ {print $3}')

$ kubectl -n kube-system describe secrets ${SECRET} | awk '/token:/{print $2}'

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtdG9rZW4tdzVocmgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWJmMTFjYzMtZjRlYi0xMWU3LTgzYWUtMDgwMDI3NjdkOWI5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZCJ9.Xuyq34ci7Mk8bI97o4IldDyKySOOqRXRsxVWIJkPNiVUxKT4wpQZtikNJe2mfUBBD-JvoXTzwqyeSSTsAy2CiKQhekW8QgPLYelkBPBibySjBhJpiCD38J1u7yru4P0Pww2ZQJDjIxY4vqT46ywBklReGVqY3ogtUQg-eXueBmz-o7lJYMjw8L14692OJuhBjzTRSaKW8U2MPluBVnD7M2SOekDff7KpSxgOwXHsLVQoMrVNbspUCvtIiEI1EiXkyCNRGwfnd2my3uzUABIHFhm0\_RZSmGwExPbxflr8Fc6bxmuz-\_jSdOtUidYkFIzvEWw2vRovPgs3MXTv59RwUw

复制token,然后贴到Kubernetes dashboard

Heapster addon

Heapster是Kubernetes社区维护的容器集群监控和分析工具。Heapster会从Kubernetes apiserver取得所有Node数据,然后再通过Node获取kubelet上的数据,最后再将所有收集到数据送到Heapster后台储存InfluxDB,最后利用Grafana抓取InfluxDB数据源来进行展示。

master1通过kubectl来建立kubernetes monitor即可:

$ export ADDON\_URL="https://kairen.github.io/files/manual-v1.8/addon"

$ wget ${ADDON\_URL}/kube-monitor.yml.conf -O kube-monitor.yml

$ kubectl apply -f kube-monitor.yml

$ kubectl -n kube-system get po,svc

NAME                                           READY     STATUS    RESTARTS   AGE

...

po/heapster-74fb5c8cdc-62xzc                   4/4       Running   0          7m

po/influxdb-grafana-55bd7df44-nw4nc            2/2       Running   0          7m



NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE

...

svc/heapster               ClusterIP   10.100.242.225   <none>        80/TCP              7m

svc/monitoring-grafana     ClusterIP   10.101.106.180   <none>        80/TCP              7m

svc/monitoring-influxdb    ClusterIP   10.109.245.142   <none>        8083/TCP,8086/TCP   7m

···

简单部署Nginx 服务

Kubernetes可以选择使用指令直接建立应用和服务,或者我们可以写YAML、JSON文件来配置,如下所示:

$ kubectl run nginx --image=nginx --port=80

$ kubectl expose deploy nginx --port=80 --type=LoadBalancer --external-ip=172.16.35.12

$ kubectl get svc,po

NAME             TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE

svc/kubernetes   ClusterIP      10.96.0.1       <none>         443/TCP        1h

svc/nginx        LoadBalancer   10.97.121.243   172.16.35.12   80:30344/TCP   22s



NAME                        READY     STATUS    RESTARTS   AGE

po/nginx-7cbc4b4d9c-7796l   1/1       Running   0          28s       192.160.57.181   ,172.16.35.12   80:32054/TCP   21s

这里type可以选择NodePort和LoadBalancer在本地裸机部署,两者差异在于NodePort只映射Host port到Container port,而LoadBalancer则继承NodePort额外映射Host target port到Container port

扩展服务数量

最后,我们可以通过以下方式来扩展服务数量:

$ kubectl scale deploy nginx --replicas=2



$ kubectl get pods -o wide

NAME                    READY     STATUS    RESTARTS   AGE       IP             NODE

nginx-158599303-0h9lr   1/1       Running   0          25s       10.244.100.5   node2

nginx-158599303-k7cbt   1/1       Running   0          1m        10.244.24.3    node1

**相关阅读**

* 技术 Kubernetes Autoscaling是如何工作的? 2018/05/07

* 技术 用户评测 | Docker管理面板系列——云帮(Rainbond 出色的k8s管理面板) 2018/05/09

* 技术 如何把应用转移到Kubernetes 2018/05/04

* 技术 Kubernetes伸缩到2500个节点中遇到的问题和解决方法 2018/04/24

* 技术 kubernetes容器网络接口(CNI) midonet网络插件的设计与实现 2017/05/04

* 技术 在生产环境使用Kuberntes一年后,我们总结了这些经验和教训 2017/02/23

* 技术 关于K8s容器集群日志收集的总结 2016/12/15

* 技术 Kubernetes集群中的高性能网络策略 2016/12/08

* 行业 开了香槟的Kubernetes并不打算放慢成功的脚步 2018/05/09

* 行业 上手kubernetes之前,你应该知道这6件事 2018/05/03

* 行业 为什么说Kubernetes是云服务的未来? 2016/10/21

**开源PaaS Rainbond提供生产就绪的Kubernetes,在线体验请注册公有云(新用户7天免费)**

原创声明,本文系作者授权云+社区发表,未经许可,不得转载。

如有侵权,请联系 yunjia_community@tencent.com 删除。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏我的小碗汤

Kubernetes-dashboard的身份认证

我们成功配置安装了kubernetes-dashboard插件,但是这里似乎来了另外一个问题:我们怎样进入到dashboard?

6332
来自专栏云计算教程系列

在你的电脑上运行Kubernetes

Kubernetes 是编配平台的首选。在开发过程中,您不妨在个人电脑上运行 Kubernetes,以便在本地启动和调试应用程序。本文提供了两种在 Mac OS...

2002
来自专栏大魏分享(微信公众号:david-share)

隆重介绍!CI/CD手下的开源界六大金刚

Jenkins 2 image based on Red Hat Enterprise Linux的镜像

3933
来自专栏小狼的世界

Minikube体验

Minikube运行要求安装有VirtualBox或VMWare Fusion,我用的是VirtualBox。

3003
来自专栏云知识学习

初识TKE中K8S的Service Account

Service account是为了方便Pod里面的进程调用Kubernetes API或其他外部服务而设计的。它与User account不同

1652
来自专栏大魏分享(微信公众号:david-share)

从一张图看Devops全流程

一、持续交付工具链全图 ? 上图源自网络。上图很清晰地列出了CD几个阶段使用的工具。 CD的工具链很长,但并不是每个模块所有工具都那么流行;换言之,我们在每个模...

1.4K9
来自专栏散尽浮华

Docker集群管理工具-Kubernetes部署记录

之前介绍了Mesos+Marathon+Zookeeper管理Docker集群平台的部署 ,本篇则说下利用Kubernetes管理Docker集群环境的部署。 ...

1.5K6
来自专栏Linux运维学习之路

Docker集群编排工具之Kubernetes(K8s)介绍、安装及使用

K8s基础原理 k8s中文社区:https://www.kubernetes.org.cn/ 简介 Kubernetes与较早的集群管理系统Mesos和YARN...

1.6K5
来自专栏Jerry的SAP技术分享

Kubernetes Helm入门指南

什么是Helm?这可不是暗黑破坏神里装备的名称:头盔,而是Kubernetes的一个包管理工具,用来简化Kubernetes应用的部署和管理。我们Helm和Ku...

1740
来自专栏小狼的世界

kubeadm安装kubernetes V1.11.1 集群

如果想要用二进制方法安装最新版本的Docker,可以参考我之前的文章在Redhat 7.3中采用离线方式安装Docker

7431

扫码关注云+社区

领取腾讯云代金券