注:本文基于k8s集群v1.16.2版本。
注:如无特殊说明,以下所有操作都是在master节点上执行。
Helm可以看作是k8s集群的包管理工具,通过Helm可以快速安装很多软件,比如mysql,nginx等,当然,也可以把自己的应用交给Helm来管理和安装。
Helm架构由Helm客户端、Helm服务端Tiller和Helm仓库Chart组成。
其中,Helm客户端可以部署在任意地方,只要能够访问k8s集群即可; Tiller服务端部署在k8s集群中。
注:为了方便,本文直接把Helm客户端安装在master节点上。
必须有一个k8s集群。
(1)下载Helm发布包
打开Helm Release,下载一个稳定版本,我这里下载的是v2.15.2版本的 Linuxamd64
。
注:这里要下载稳定版本,不要下载RC(Release Candidate)版本,因为我们后面要修改镜像,非稳定版本国内的镜像仓库不一定有。
下载下来之后,解压,然后把helm可执行文件拷贝到/usr/local/bin目录下,之后就可以执行helm命令了。
[root@instance-ji0xk9uh-1 ~]# ls
calico-3.9.2.yaml helm-v2.15.2-linux-amd64.tar.gz init_master.sh install_kubelet.sh kubeadm-config.yaml nginx-ingress.yaml
[root@instance-ji0xk9uh-1 ~]# tar zxvf helm-v2.15.2-linux-amd64.tar.gz
linux-amd64/
linux-amd64/tiller
linux-amd64/helm
linux-amd64/README.md
linux-amd64/LICENSE
[root@instance-ji0xk9uh-1 ~]# ls
calico-3.9.2.yaml helm-v2.15.2-linux-amd64.tar.gz init_master.sh install_kubelet.sh kubeadm-config.yaml linux-amd64 nginx-ingress.yaml
[root@instance-ji0xk9uh-1 ~]# cp linux-amd64/helm /usr/local/bin/
[root@instance-ji0xk9uh-1 ~]# helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Error: could not find tiller
直接执行 helm init
即可。
[root@master ~]# helm init
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
检查是否安装成功, heml version
。
[root@master ~]# helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Error: could not find a ready tiller pod
发现并没有安装成功,再看看pod的状态, kubectlgetpod-n kube-system
。
[root@master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
# 省略...
tiller-deploy-58f57c5787-v6z7d 0/1 ImagePullBackOff 0 5m50s
发现pod的状态为 ImagePullBackOff
,出现这个的原因很大可能是镜像没有拉取下来。
我们使用 describe
查看更详细的原因。
[root@master ~]# kubectl describe pod/tiller-deploy-58f57c5787-v6z7d -n kube-system
# 省略...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/tiller-deploy-58f57c5787-v6z7d to node1
Normal Pulling 2m19s (x4 over 4m23s) kubelet, node1 Pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2"
Warning Failed 2m4s (x4 over 4m7s) kubelet, node1 Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.15.2": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 2m4s (x4 over 4m7s) kubelet, node1 Error: ErrImagePull
Normal BackOff 97s (x6 over 4m7s) kubelet, node1 Back-off pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2"
Warning Failed 84s (x7 over 4m7s) kubelet, node1 Error: ImagePullBackOff
从这里可以看到它使用的镜像是 gcr.io/kubernetes-helm/tiller:v2.15.2
,我们直接在镜像仓库搜索这个镜像。
[root@master ~]# docker search gcr.io/kubernetes-helm/tiller:v2.15.2
Error response from daemon: invalid registry endpoint https://gcr.io/v1/: Get https://gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry gcr.io` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/gcr.io/ca.crt
从上面的日志可以看到是超时了,这时候我们换一个角度,看看我们的镜像仓库有没有 tiller
的镜像。
[root@master ~]# docker search tiller
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
jessestuart/tiller Nightly multi-architecture (amd64, arm64, ar… 16 [OK]
sapcc/tiller Mirror of https://gcr.io/kubernetes-helm/til… 8
ist0ne/tiller https://gcr.io/kubernetes-helm/tiller 3 [OK]
rancher/tiller 2
jmgao1983/tiller from gcr.io/kubernetes-helm/tiller 2 [OK]
ibmcom/tiller Docker Image for IBM Cloud private-CE (Commu… 1
itinerisltd/tiller 1
luxas/tiller 1
ansibleplaybookbundle/tiller-apb An APB that deploys tiller for use with helm. 1 [OK]
cfplatformeng/tiller 0
cfplatformeng/tiller-ubuntu 0
ibmcom/tiller-ppc64le Docker Image for IBM Cloud Private-CE (Commu… 0
kubeapps/tiller-proxy A web-based UI for deploying and managing ap… 0
kubeapps/tiller-proxy-ci Store temporary images generated by CI system 0
sorenmat/tiller 0
cgetzen/tiller Custom Tiller Tests 0
appscode/tiller 0
fkpwolf/tiller 0
itinerisltd/tiller-circleci 0
anjia0532/tiller 0
pcanham/tiller tiller image for Raspberry Pi for testing He… 0
kontenapharos/tiller 0
renaultdigital/tillerless-helm-gcloud Add tillerless plugin to helm-gcloud image 0
4admin2root/tiller gcr.io/kubernetes-helm/tiller 0 [OK]
zhaowenlei/tiller FROM gcr.io/kubernetes-helm/tiller:TAG 0
可以查到很多tiller的镜像,我们这里选第二个,看它的描述,是 gcr.io/kubernetes-helm/tiller
的镜像。
我们尝试拉取看看,注意这里一定要拉取跟上面安装的helm相同版本的镜像。
[root@master ~]# docker pull sapcc/tiller:v2.15.2
v2.15.2: Pulling from sapcc/tiller
89d9c30c1d48: Pull complete
45d707e102b0: Pull complete
d20e148dce16: Pull complete
8ca326d5a0c5: Pull complete
Digest: sha256:4554a65fb8278d93f1c2c1f335ddbfcd6faa016c24b97e8de46c6b8fc1e9e7f5
Status: Downloaded newer image for sapcc/tiller:v2.15.2
可以看到拉取成功了。所以,我们换一种思路,把tiller安装的yaml中使用的镜像改成上面这个可用的镜像,那么要怎么修改呢?
其实,也很简单,k8s提供了命令可以直接修改现在正在运行的pod的yaml配置, kubectl edit pod xxx
。
注,这里也可以修改deployment的配置,这样更持久,因为pod是由deployment创建的。
[root@master ~]# kubectl edit pod tiller-deploy-58f57c5787-v6z7d -n kube-system
#省略...
spec:
automountServiceAccountToken: true
containers:
- env:
- name: TILLER_NAMESPACE
value: kube-system
- name: TILLER_HISTORY_MAX
value: "0"
image: sapcc/tiller:v2.15.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /liveness
port: 44135
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: tiller
#省略...
找到image字段,把后面的镜像修改为 sapcc/tiller:v2.15.2
。
然后,pod会自动更新,过一会再次查看pod的状态。
[root@master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
#省略...
tiller-deploy-58f57c5787-v6z7d 1/1 Running 0 25m
可以看到tiller运行起来了,再次使用 helm version
检查结果。
[root@master ~]# helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
至此,说明tiller安装成功了。
helm search mysql,查找mysql的chart。
helm install mysql,安装mysql。
[root@master ~]# helm search mysql
NAME CHART VERSION APP VERSION DESCRIPTION
stable/mysql 1.4.0 5.7.27 Fast, reliable, scalable, and easy to use open-source rel...
stable/mysqldump 2.6.0 2.4.1 A Helm chart to help backup MySQL databases using mysqldump
stable/prometheus-mysql-exporter 0.5.2 v0.11.0 A Helm chart for prometheus mysql exporter with cloudsqlp...
stable/percona 1.2.0 5.7.17 free, fully compatible, enhanced, open source drop-in rep...
stable/percona-xtradb-cluster 1.0.3 5.7.19 free, fully compatible, enhanced, open source drop-in rep...
stable/phpmyadmin 4.1.1 4.9.1 phpMyAdmin is an mysql administration frontend
stable/gcloud-sqlproxy 0.6.1 1.11 DEPRECATED Google Cloud SQL Proxy
stable/mariadb 6.12.2 10.3.18 Fast, reliable, scalable, and easy to use open-source rel...
[root@master ~]# helm install stable/mysql
Error: no available release name found
又报错了,经查找相关资料,是说k8s集群1.16.x版本加入了RBAC权限相关的东西,必须给tiller定义一个service account。
OK,使用以下脚本,先卸载tiller(删除相应的deployment),再设置权限,重新安装。
注,如果安装tiller过程中出现如下错误也是执行下面的脚本。
Error:error installing:the server couldnotfind the requested resource
重装tiller脚本。
# 先卸载tiller,删除其deployment即可
kubectl delete deployment.apps/tiller-deploy -n kube-system
# 创建serviceaccount
kubectl create serviceaccount --namespace kube-system tiller
# 绑定
kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
# 重新初始化tiller
helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -
执行结果如下:
[root@master ~]# kubectl get deployment -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
calico-kube-controllers 1/1 1 1 22h
coredns 2/2 2 2 22h
kuboard 1/1 1 1 21h
tiller-deploy 1/1 1 1 7m21s
[root@master ~]# kubectl delete deployment.apps/tiller-deploy -n kube-system
deployment.apps "tiller-deploy" deleted
[root@master ~]# kubectl create serviceaccount --namespace kube-system tiller
ec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -serviceaccount/tiller created
[root@master ~]# kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
[root@master ~]# helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -
deployment.apps/tiller-deploy created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
service/tiller-deploy configured
这时候又会出现 ImagePullBackOff
错误,同样地,重新修改tiller的yaml文件中的镜像,会自动更新。
重新安装后,再重新安装mysql。
[root@master ~]# helm install stable/mysql
NAME: bumptious-fish
LAST DEPLOYED: Thu Oct 31 15:21:43 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
bumptious-fish-mysql-test 1 0s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
bumptious-fish-mysql 0/1 1 0 0s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
bumptious-fish-mysql Pending 0s Filesystem
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
bumptious-fish-mysql-68bf985647-649ms 0/1 Pending 0 0s
==> v1/Secret
NAME TYPE DATA AGE
bumptious-fish-mysql Opaque 2 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bumptious-fish-mysql ClusterIP 10.96.43.224 <none> 3306/TCP 0s
NOTES:
MySQL can be accessed via port 3306 on the following DNS name from within your cluster:
bumptious-fish-mysql.default.svc.cluster.local
To get your root password run:
MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)
To connect to your database:
1. Run an Ubuntu pod that you can use as a client:
kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il
2. Install the mysql client:
$ apt-get update && apt-get install mysql-client -y
3. Connect using the mysql cli, then provide your password:
$ mysql -h bumptious-fish-mysql -p
To connect to your database directly from outside the K8s cluster:
MYSQL_HOST=127.0.0.1
MYSQL_PORT=3306
# Execute the following command to route the connection:
kubectl port-forward svc/bumptious-fish-mysql 3306
mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}
查看mysql是否启动成功,我们是安装到默认命名空间下的,所以不需要带 -n
参数了。
[root@master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
bumptious-fish-mysql-68bf985647-649ms 0/1 Pending 0 40s
会发现一直是在 Pending
状态,同样地,使用 kubectl describe
命令查看详细的错误。
[root@master ~]# kubectl describe pod bumptious-fish-mysql-68bf985647-649ms
# 省略...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
看最后两行,是说pod没有绑定到PersistentVolumeClaims(简写pvc),我们再查查pvc。
[root@master ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
bumptious-fish-mysql Pending 2m18s
可以看到pvc也是 Pending
状态,同样地,describe一下。
[root@master ~]# kubectl describe pvc bumptious-fish-mysql
# 省略...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 11s (x12 over 2m44s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
看最后一行,说是没有可用的persistent volume(简称pv)且没有设置storage class,这又是些什么东西呢。
这里就牵涉到k8s的整体架构了,针对有状态的pod,需要先关联一个pvc,pvc会与具体的pv关联,pv可以理解成是磁盘上的一块空间,或者是远程存储等。
关于pv的自动创建也有很多方案,比如nfs等,为了不增加复杂性,我们这里采用手动创建两块pv给mysql使用。
手动创建pv的脚本如下:
kubectl create -f - <<EOF
kind: PersistentVolume
apiVersion: v1
metadata:
name: [名称]
labels:
type: local
spec:
capacity:
storage: [使用的空间]
accessModes:
- ReadWriteOnce
hostPath:
path: "存储的路径"
EOF
执行结果如下:
[root@master ~]# kubectl create -f - <<EOF
> kind: PersistentVolume
> apiVersion: v1
> metadata:
> name: mysql-pv
> labels:
> type: local
> spec:
> capacity:
> storage: 10Gi
> accessModes:
> - ReadWriteOnce
> hostPath:
> path: "/data/storage/mysql"
> EOF
persistentvolume/mysql-pv created
[root@master ~]# kubectl get pv,pvc,pod
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/mysql-pv 10Gi RWO Retain Bound default/bumptious-fish-mysql 67s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/bumptious-fish-mysql Bound mysql-pv 10Gi RWO 5m49s
NAME READY STATUS RESTARTS AGE
pod/bumptious-fish-mysql-68bf985647-649ms 1/1 Running 0 5m49s
这里的 kubectlgetpv,pvc,pod
要过一会再执行,或者前面加一个watch,即 watch kubectlgetpv,pvc,pod
,直到pod的状态变成Running且READY为1/1的时候表示mysql安装成功了。
到这里,基本可以确定mysql是安装成功了,但是没有连接进去谁又敢百分百说安装成功了呢,OK,我们下面尝试连接到mysql中。
查看mysql的service。
[root@master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bumptious-fish-mysql ClusterIP 10.96.43.224 <none> 3306/TCP 9m36s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
可以看到服务暴露的方式为 ClusterIP
,我们把它修改为 NodePort
,NodePort可以让我们从外部访问服务。
[root@master ~]# kubectl edit svc bumptious-fish-mysql
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-10-31T07:21:43Z"
labels:
app: bumptious-fish-mysql
chart: mysql-1.4.0
heritage: Tiller
release: bumptious-fish
name: bumptious-fish-mysql
namespace: default
resourceVersion: "125451"
selfLink: /api/v1/namespaces/default/services/bumptious-fish-mysql
uid: 70be1da0-a10f-4439-9d3e-fd516f8785bf
spec:
clusterIP: 10.96.43.224
externalTrafficPolicy: Cluster
ports:
- name: mysql
nodePort: 32684
port: 3306
protocol: TCP
targetPort: mysql
selector:
app: bumptious-fish-mysql
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
找到spec.type,将其值修改为 NodePort
,保存退出,查看自动生成的端口。
[root@master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bumptious-fish-mysql NodePort 10.96.43.224 <none> 3306:32684/TCP 18h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 41h
可以看到生成的端口为 32684
,再查看mysql的初始密码,命令在安装mysql的时候提示过,注意这里拷贝你安装时提示的命令。
[root@master ~]# echo $(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)
3XvXkqckUE
在外部使用mysql客户端连接(当然,也可以在集群内部装一个mysql客户端,这样就不需要暴露外网ip及端口了):
ip为任意worker节点的公网ip;
端口为上面获取到的32684;
用户名为root,密码为上面获取到的3XvXkqckUE;
C:\> mysql -h[xx.xx.xx.xx] -P32684 -uroot -p3XvXkqckUE
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13658
Server version: 5.7.14 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.04 sec)
mysql> create database test;
Query OK, 1 row affected (0.04 sec)
mysql> use test;
Database changed
mysql> create table user(id int, name varchar(50));
Query OK, 0 rows affected (0.05 sec)
mysql> insert into user values(1,"hello");
Query OK, 1 row affected (0.05 sec)
mysql> select * from user;
+------+-------+
| id | name |
+------+-------+
| 1 | hello |
+------+-------+
1 row in set (0.04 sec)
成功连接进去了,说明mysql安装成功了。
本章成功安装了Helm,并使用Helm成功安装了mysql,这里有几个需要注意的点。
(1)k8s集群v1.16.x版本需要设置RBAC权限,才能成功安装Helm;
(2)网络问题导致无法成功拉取tiller镜像,所以需要手动修改tiller pod的yaml文件;
(3)mysql安装需要创建持久卷PV,我们这里使用手动创建的方式。
官方文档:https://helm.sh/docs/using_helm
本文在官方文档的基础上补充了一些异常的处理。