二进制安装k8s集群(16)-安装nginx-ingress-controller
二进制安装k8s集群(16)-安装nginx-ingress-controller
在上一篇文章里我们主要介绍安装k8s集群内的基础服务kube-dashboard，这里我们继续介绍安装k8s集群内基础服务nginx-ingress，这个基础服务也创建在kube-system namesapce里，是以deployment的方式运行。当然 daemonset也是可以的，这里没有硬性要求。image镜像从我们的private repo pull(以前文章里介绍过harbor private repo的创建，以及镜像的push和pull)。当然原始image来源于官方的quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1，不过要下载需要访问国外网站。另外对于ingress方案，一般有nginx-ingress，traefik ingress(traefik2.0也已经问世了，都是可以选择的)，haproxy ingress等，实际情况用哪种请根据团队和实际的需求来选择。
创建配置文件目录：
由于nginx-ingress-controller是以deployment的方式部署在k8s集群里的，一般都会有yaml部署文件，目前都放在此目录里。
mkdir -p /opt/application/k8s/nginx-ingress
cd /opt/application/k8s/nginx-ingress
创建nginx-ingress的service-account：
nginx-ingress-controlle需要访问kube-apiserver来得到集群中的对象资源，从而展示在UI上。对于资源访问，k8s有自己的策略，这里给nginx-ingress-controlle创建单独的service-account，cluster-role-binding。目前就不详细展开了，有兴趣的同学可以看一下k8s的RBAC访问策略。
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: serviceaccount-nginx-ingress
namespace: kube-system
imagePullSecrets:
- name: container-registry
EOF
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-service-account.yaml
kubectl describe serviceaccount serviceaccount-nginx-ingress -n kube-system
创建nginx-ingress的配置configmap：
nginx-ingress-controller也有自己的配置，我们把它的配置创建在k8s configmap资源里，然后在容器里挂载这个configmap的数据，从而提供给nginx-ingress-controller配置。当然配置项比较多，这里就不逐一介绍，有兴趣的同学请参考nginx-ingress的配置文档。
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/cli-arguments.md
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-nginx-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-nginx-configuration
namespace: kube-system
EOF
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-tcp-service-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-tcp-services
namespace: kube-system
EOF
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-udp-service-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-udp-services
namespace: kube-system
EOF
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-nginx-config-map.yaml
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-tcp-service-config-map.yaml
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-udp-service-config-map.yaml
kubectl describe configmap configmap-nginx-configuration -n kube-system
kubectl describe configmap configmap-tcp-services -n kube-system
kubectl describe configmap configmap-udp-services -n kube-system
创建nginx-ingress的cluster-role-binding：
这里为了方便，我们绑定k8s集群内置的cluster-admin账号。注意账户cluster-admin的权限比较大(从名字里就可以看出来)，对于实际应用请根据自身需求看是否创建cluster-role或者role来绑定给service-account。
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-cluster-role-binding.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-role-binding-nginx-ingress
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: serviceaccount-nginx-ingress
namespace: kube-system
EOF
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-cluster-role-binding.yaml
kubectl describe clusterrolebinding cluster-role-binding-nginx-ingress -n kube-system
创建nginx-ingress的证书secret：
nginx-ingress-controller负责向外暴露集群内部的service，我们这里开启ssl，给nginx-ingress-controller配置默认ssl证书(在创建ingress rule的时候同样也可以指定ssl证书，这里是在没有指定ingress rule ssl的情况下默认ssl配置)。这里的证书以secret的形式存储在k8s里，然后挂载给nginx-ingress-controller容器作为配置使用。当然请提前制作好相关证书，可以参考以前文章里制作docker server证书。
kubectl create secret tls nginx-ingress-cert-secret --namespace=kube-system \
--cert=/opt/sw/cert/k8s-ingress/tls.crt \
--key=/opt/sw/cert/k8s-ingress/tls.key
kubectl describe secret nginx-ingress-cert-secret -n kube-system
创建nginx-ingress的kubeconfig文件secret：
nginx-ingress-controller需要访问kube-apiserver来得到集群中的对象资源，用以向外暴露服务。我们kube-apiserver的访问需要kubeconfig文件。这里我们提前制作好(可以参考以前文章里安装kubectl的时候制作kubeconfig文件)，然后以secret的形式存储在k8s里，挂载给nginx-ingress-controller容器作为配置使用。
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-cluster k8s-cluster-one --server=https://172.20.11.41:6443 --certificate-authority=/etc/kubernetes/kubeconfig/cert/ca.crt --embed-certs=true
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-credentials k8s-access-user-with-cert --client-certificate=/etc/kubernetes/kubeconfig/cert/k8sapiserver-client.crt --client-key=/etc/kubernetes/kubeconfig/cert/k8sapiserver-client.key --embed-certs=true
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-context context-one --cluster=k8s-cluster-one --user=k8s-access-user-with-cert
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert use-context context-one
kubectl create secret generic nginx-ingress-apiserver-access-secret --namespace=kube-system \
--from-file=kubeconfig=/etc/kubernetes/kubeconfig/config-contains-cert
kubectl describe secret nginx-ingress-apiserver-access-secret -n kube-system
创建nginx-ingress的deployment：
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-nginx-ingress
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
k8s-app: nginx-ingress
template:
metadata:
labels:
k8s-app: nginx-ingress
spec:
serviceAccountName: serviceaccount-nginx-ingress
containers:
- name: nginx-ingress
image: 172.20.11.41:1034/infra/nginx-ingress-controller:0.25.1
imagePullPolicy: IfNotPresent
args:
- /nginx-ingress-controller
- --configmap=kube-system/configmap-nginx-configuration
- --tcp-services-configmap=kube-system/configmap-tcp-services
- --udp-services-configmap=kube-system/configmap-udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
- --http-port=80
- --https-port=443
- --healthz-port=10254
- --health-check-path=/healthz
- --health-check-timeout=15
- --alsologtostderr
- --default-ssl-certificate=kube-system/nginx-ingress-cert-secret
- --kubeconfig=/config/kubeconfig
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
volumeMounts:
- name: kube-apiserver-access-config
mountPath: /config
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
volumes:
- name: kube-apiserver-access-config
secret:
secretName: nginx-ingress-apiserver-access-secret
EOF
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-deployment.yaml
kubectl describe deployment deployment-nginx-ingress -n kube-system
创建nginx-ingress的service：
这里把service定义成node-port类型，以node port的方式提供给外部访问。node port定义为8443(https)和8080(http)，同时也请打开8443/8080的防火墙端口(可以参考以前文章如何打开某个端口)。
cat > /opt/application/k8s/nginx-ingress/nginx-ingress-service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: service-nginx-ingress
namespace: kube-system
spec:
selector:
k8s-app: nginx-ingress
type: NodePort
ports:
- name: nginx-ingress-https
nodePort: 8443
port: 443
targetPort: 443
protocol: TCP
- name: nginx-ingress-http
nodePort: 8080
port: 80
targetPort: 80
protocol: TCP
EOF
kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-service.yaml
kubectl describe service service-nginx-ingress -n kube-system
查看nginx-ingress在集群中的pod：
kubectl get pods -n kube-system|grep nginx