1、K8s为什么要使用存储?
答:k8s中的副本控制器保证了pod的始终存储,却保证不了Pod中的数据。只有启动一个新pod的,之前pod中的数据会随着容器的删掉而丢失。k8s中的rc启动指定数量的Pod,当某个Pod死掉了,会在新的节点启动新的Pod,k8s中想要实现数据持久化,需要使用一个叫做共享存储的,让Pod里面的数据挂载到这个共享存储上面,就算在新的节点启动新的Pod,依然可以保证数据不丢失。
2、k8s中的PV和PVC的概念。
答:PersistentVolume(简称为PV,持久化存储),由管理员添加的一个存储的描述,是一个全局资源,没有namespace的限制,包含存储的类型,存储的大小和访问模式等等。它的生命周期独立于Pod,例如当使用它的Pod销毁时对PV没有影响。 PersistentVolumeClaim(简称为PVC),是Namespace里面的资源,描述对PV的一个请求。请求信息包含存储大小,访问模式等等。注意,PV和PVC是一一绑定的。
3、首先这里创建PV使用到的是NFS,这里首先安装一下NFS,如下所示:
需要Master主节点先安装NFS服务端,在所有Node节点安装NFS客户端,不然Pod无法使用,因为挂载不上,在三台机器都执行下面的命令即可。
1 [root@k8s-master ~]# yum install nfs-utils.x86_64
2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
3
4 This system is not registered with an entitlement server. You can use subscription-manager to register.
5
6 Loading mirror speeds from cached hostfile
7 * base: mirrors.tuna.tsinghua.edu.cn
8 * extras: mirrors.bfsu.edu.cn
9 * updates: mirrors.bfsu.edu.cn
10 base | 3.6 kB 00:00:00
11 extras | 2.9 kB 00:00:00
12 updates | 2.9 kB 00:00:00
13 Resolving Dependencies
14 --> Running transaction check
15 ---> Package nfs-utils.x86_64 1:1.3.0-0.61.el7 will be updated
16 ---> Package nfs-utils.x86_64 1:1.3.0-0.66.el7 will be an update
17 --> Finished Dependency Resolution
18
19 Dependencies Resolved
20
21 =================================================================================================================================================================================================================
22 Package Arch Version Repository Size
23 =================================================================================================================================================================================================================
24 Updating:
25 nfs-utils x86_64 1:1.3.0-0.66.el7 base 412 k
26
27 Transaction Summary
28 =================================================================================================================================================================================================================
29 Upgrade 1 Package
30
31 Total size: 412 k
32 Is this ok [y/d/N]: y
33 Downloading packages:
34 Running transaction check
35 Running transaction test
36 Transaction test succeeded
37 Running transaction
38 Updating : 1:nfs-utils-1.3.0-0.66.el7.x86_64 1/2
39 Cleanup : 1:nfs-utils-1.3.0-0.61.el7.x86_64 2/2
40 Verifying : 1:nfs-utils-1.3.0-0.66.el7.x86_64 1/2
41 Verifying : 1:nfs-utils-1.3.0-0.61.el7.x86_64 2/2
42
43 Updated:
44 nfs-utils.x86_64 1:1.3.0-0.66.el7
45
46 Complete!
47 [root@k8s-master ~]#
在三台机器都要安装nfs-utils的包的,然后在服务器端,开始配置配置文件/etc/exports,如下所示:
1 [root@k8s-master ~]# vim /etc/exports
配置内容,如下所示:
1 # 共享data目录,允许192.168.110.*访问,即允许110段ip地址访问。读写权限,同步,不做root用户、所有用户的UID映射
2 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash)
创建/data/目录下面的一个目录,如下所示:
1 [root@k8s-master ~]# cat /etc/exports
2 # 共享data目录,允许192.168.110.*访问,即允许110段ip地址访问。读写权限,同步,不做root用户、所有用户的UID映射
3 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash)
4
5
6 [root@k8s-master ~]# mkdir /data/k8s
7 mkdir: cannot create directory ‘/data/k8s’: No such file or directory
8 [root@k8s-master ~]# mkdir /data/k8s -p
9 [root@k8s-master ~]#
然后重启nfs、rpcbind,如下所示:
1 [root@k8s-master ~]# systemctl restart rpcbind.service
2 [root@k8s-master ~]# systemctl restart nfs.service
3 [root@k8s-master ~]#
在另外两个节点查看是否可以查看到nfs,如下所示:
1 [root@k8s-node2 ~]# showmount -e 192.168.110.133
2 Export list for 192.168.110.133:
3 /data 192.168.110.0/24
4 [root@k8s-node2 ~]#
5
6
7 [root@k8s-node3 ~]# showmount -e 192.168.110.133
8 Export list for 192.168.110.133:
9 /data 192.168.110.0/24
10 [root@k8s-node3 ~]#
下面开始创建一个PV,如下所示:
1 [root@k8s-master ~]# cd k8s/
2 [root@k8s-master k8s]# ls
3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip
4 [root@k8s-master k8s]# mkdir volume
5 [root@k8s-master k8s]# cd volume/
6 [root@k8s-master volume]# vim test-pv.yaml
配置内容,如下所示:
1 apiVersion: v1
2 kind: PersistentVolume
3 metadata:
4 name: testpv
5 labels:
6 type: testpv
7 spec:
8 capacity:
9 storage: 10Gi
10 accessModes:
11 - ReadWriteMany
12 persistentVolumeReclaimPolicy: Recycle
13 nfs:
14 path: "/data/k8s"
15 server: 192.168.110.133
16 readOnly: false
开始创建这个PV,并进行查看,如下所示:
1 [root@k8s-master volume]# kubectl create -f test-pv.yaml
2 persistentvolume "testpv" created
3 [root@k8s-master volume]# kubectl get pv -o wide
4 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
5 testpv 10Gi RWX Recycle Available 10s
6 [root@k8s-master volume]#
再创建一个5Gi的PV,修改一下配置文件,然后进行创建,名称必须不一样,标签可以一样的,如下所示:
1 [root@k8s-master volume]# vim test-pv.yaml
2 [root@k8s-master volume]# kubectl create -f test-pv.yaml
3 persistentvolume "testpv2" created
4 [root@k8s-master volume]# kubectl get pv -o wide
5 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
6 testpv 10Gi RWX Recycle Available 3m
7 testpv2 5Gi RWX Recycle Available 3s
8 [root@k8s-master volume]#
开始创建PVC,如下所示:
1 apiVersion: v1
2 kind: PersistentVolumeClaim
3 kind: PersistentVolumeClaim
4 apiVersion: v1
5 metadata:
6 name: nfs
7 spec:
8 accessModes:
9 - ReadWriteMany
10 resources:
11 requests:
12 storage: 1Gi
1 [root@k8s-master volume]# kubectl create -f test-pvc.yaml
2 persistentvolumeclaim "nfs" created
3 [root@k8s-master volume]# kubectl get pvc -o wide
4 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
5 nfs Bound testpv2 5Gi RWX 9s
6 [root@k8s-master volume]#
此时,可以发现pvc已经绑定到了testpv2这个PV上面了,pvc优先绑定到容量小的pv上面的。自己也可以再创建一个6G的PV,然后再创建一个7G的PVC,可以发现这个PVC会绑定到10G的这个PV上面了的。
4、K8s持久化实战
使用k8s运行Java Web项目,这里紧接着这个链接的操作,创建一个mysql,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl create -f mysql-rc.yml
2 replicationcontroller "mysql" created
3 [root@k8s-master tomcat_demo]# kubectl create -f mysql-svc.yml
4 service "mysql" created
5 [root@k8s-master tomcat_demo]# kubectl get all -o wide
6 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
7 rc/mysql 1 1 1 14s mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
8
9 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
10 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
11 svc/mysql 10.254.234.52 <none> 3306/TCP 11s app=mysql
12
13 NAME READY STATUS RESTARTS AGE IP NODE
14 po/mysql-4j7qk 1/1 Running 0 14s 172.16.16.3 k8s-node3
15 [root@k8s-master tomcat_demo]#
下面是mysql-rc.yaml的配置文件,如下所示:
1 [root@k8s-master tomcat_demo]# cat mysql-rc.yml
2 apiVersion: v1
3 kind: ReplicationController
4 metadata:
5 name: mysql
6 spec:
7 replicas: 1
8 selector:
9 app: mysql
10 template:
11 metadata:
12 labels:
13 app: mysql
14 spec:
15 containers:
16 - name: mysql
17 image: 192.168.110.133:5000/mysql:5.7.30
18 ports:
19 - containerPort: 3306
20 env:
21 - name: MYSQL_ROOT_PASSWORD
22 value: '123456'
23 [root@k8s-master tomcat_demo]#
下面是mysql-svc.yaml的配置文件,如下所示:
1 [root@k8s-master tomcat_demo]# cat mysql-svc.yml
2 apiVersion: v1
3 kind: Service
4 metadata:
5 name: mysql
6 spec:
7 ports:
8 - port: 3306
9 targetPort: 3306
10 selector:
11 app: mysql
12 [root@k8s-master tomcat_demo]#
下面是tomcat-rc.yml的配置文件,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 11m myweb 192.168.110.133:5000/tomcat-book:latest app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
8 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql
9 svc/myweb 10.254.14.249 <nodes> 8080:30008/TCP 10m app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3
13 po/myweb-f2dqn 1/1 Running 0 11m 172.16.16.5 k8s-node3
14 [root@k8s-master tomcat_demo]# vim tomcat-rc.yml
15 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml
16 apiVersion: v1
17 kind: ReplicationController
18 metadata:
19 name: myweb
20 spec:
21 replicas: 1
22 selector:
23 app: myweb
24 template:
25 metadata:
26 labels:
27 app: myweb
28 spec:
29 containers:
30 - name: myweb
31 # image: 192.168.110.133:5000/tomcat:latest
32 image: 192.168.110.133:5000/tomcat-book:latest
33 ports:
34 - containerPort: 8080
35 env:
36 - name: MYSQL_SERVICE_HOST
37 # value: '192.168.110.133' # 如果已经安装了dns服务,那么这里可以直接使用mysql的svc即Service的名称mysql
38 value: '10.254.234.52'
39 - name: MYSQL_SERVICE_PORT
40 value: '3306'
41 [root@k8s-master tomcat_demo]#
下面是tomcat-svc.yml的配置文件,如下所示:
1 [root@k8s-master tomcat_demo]# cat tomcat-svc.yml
2 apiVersion: v1
3 kind: Service
4 metadata:
5 name: myweb
6 spec:
7 type: NodePort
8 ports:
9 - port: 8080
10 nodePort: 30008
11 selector:
12 app: myweb
13 [root@k8s-master tomcat_demo]#
然后可以通过Navicat连上这个机器的Mysql,然后创建一个数据库,将sql执行一下,就完成了Mysql的任务了,下面就是将web项目搞到tomcat里面,然后使用k8s启动tomcat就行了。
怎么创建java web端的镜像呢,首先镜像要运行在tomcat里面的,创建镜像的第一步,说明继承自那个基础镜像,因为程序是Java web程序,肯定是需要用到tomcat的,先将tomcat的镜像下载下来。
1 [root@k8s-master tomcat_demo]# vim Dockerfile
配置内容,如下所示:
1 # 第一步,说明继承那个基础镜像,tomcat作为基础镜像。此处告诉Docker我要做自己的镜像,以tomcat镜像为起点。
2 from docker.io/tomcat:latest3
4 # 第二步,描述镜像的所有者,可以省略。
5 MAINTAINER biehl biehl@qq.com
6
7 # 第三步,将自己的web应用放到镜像里面,将本地的war包拷贝到tomcat的webapps目录下面。可以看看程序是否可以随着tomcat启动起来。
8 COPY book-master.war /usr/local/tomcat/webapps
dokcer build命令就是用来执行Dockerfile里面所描述的每一件事情,最终会把docker镜像给我们构建出来。
1 [root@k8s-master tomcat_demo]# docker build -t tomcat-book:latest .
2 Sending build context to Docker daemon 9.699 MB
3 Step 1/3 : FROM 192.168.110.133:5000/tomcat:latest
4 ---> 2eb5a120304e
5 Step 2/3 : MAINTAINER biehl biehl@qq.com
6 ---> Running in a51d06f7bddb
7 ---> 0be82c0d264e
8 Removing intermediate container a51d06f7bddb
9 Step 3/3 : COPY book-master.war /usr/local/tomcat/webapps
10 ---> 98fbba0fd03c
11 Removing intermediate container e71c8a9f14c9
12 Successfully built 98fbba0fd03c
13 [root@k8s-master tomcat_demo]# docker images
14 REPOSITORY TAG IMAGE ID CREATED SIZE
15 book-master.war latest 98fbba0fd03c 11 seconds ago 657 MB
然后可以将这个上传到私有仓库里面,方便其他Node节点下载,如下所示:
1 [root@k8s-master ~]# docker tag tomcat-book:latest 192.168.110.133:5000/tomcat-book:latest
2 [root@k8s-master ~]# docker push 192.168.110.133:5000/tomcat-book:latest
然后创建tomcat的rc、svc,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-rc.yml
2 replicationcontroller "myweb" created
3 [root@k8s-master tomcat_demo]# kubectl get all -o wide
4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
5 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
6 rc/myweb 1 1 1 7s myweb 192.168.110.133:5000/tomcat-book:latest app=myweb
7
8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
9 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
10 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql
11
12 NAME READY STATUS RESTARTS AGE IP NODE
13 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3
14 po/myweb-cvd16 1/1 Running 0 7s 172.16.16.5 k8s-node3
15 [root@k8s-master tomcat_demo]# kubectl describe pod myweb-cvd16
16 Name: myweb-cvd16
17 Namespace: default
18 Node: k8s-node3/192.168.110.135
19 Start Time: Mon, 29 Jun 2020 20:12:54 +0800
20 Labels: app=myweb
21 Status: Running
22 IP: 172.16.16.5
23 Controllers: ReplicationController/myweb
24 Containers:
25 myweb:
26 Container ID: docker://1caf37bb344e394af5701035f9a28631129260f4cb5a5f1f88a214a66891631e
27 Image: 192.168.110.133:5000/tomcat-book:latest
28 Image ID: docker-pullable://192.168.110.133:5000/tomcat-book@sha256:e838afdd7d378bd078dbf431d5f956a197b94caba624b238049108e64cf66526
29 Port: 8080/TCP
30 State: Running
31 Started: Mon, 29 Jun 2020 20:12:55 +0800
32 Ready: True
33 Restart Count: 0
34 Volume Mounts: <none>
35 Environment Variables:
36 MYSQL_SERVICE_HOST: 10.254.234.52
37 MYSQL_SERVICE_PORT: 3306
38 Conditions:
39 Type Status
40 Initialized True
41 Ready True
42 PodScheduled True
43 No volumes.
44 QoS Class: BestEffort
45 Tolerations: <none>
46 Events:
47 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
48 --------- -------- ----- ---- ------------- -------- ------ -------
49 22s 22s 1 {default-scheduler } Normal Scheduled Successfully assigned myweb-cvd16 to k8s-node3
50 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Pulling pulling image "192.168.110.133:5000/tomcat-book:latest"
51 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Pulled Successfully pulled image "192.168.110.133:5000/tomcat-book:latest"
52 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Created Created container with docker id 1caf37bb344e; Security:[seccomp=unconfined]
53 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Started Started container with docker id 1caf37bb344e
54 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-svc.yml
55 service "myweb" created
56 [root@k8s-master tomcat_demo]# kubectl get all -o wide
57 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
58 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
59 rc/myweb 1 1 1 34s myweb 192.168.110.133:5000/tomcat-book:latest app=myweb
60
61 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
62 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
63 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql
64 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 4s app=myweb
65
66 NAME READY STATUS RESTARTS AGE IP NODE
67 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3
68 po/myweb-cvd16 1/1 Running 0 34s 172.16.16.5 k8s-node3
69 [root@k8s-master tomcat_demo]#
访问地址http://192.168.110.133:30008/,报错了,源服务器未能找到目标资源的表示或者是不愿公开一个已经存在的资源表示。
还按照之前搞过的,这次找到是那台机器运行了这个Pod,找到了,然后将webapps.dist目录里面的文件拷贝到webapps里面就可以了。
1 [root@k8s-node3 ~]# docker ps
2 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3 1caf37bb344e 192.168.110.133:5000/tomcat-book:latest "catalina.sh run" 6 minutes ago Up 6 minutes k8s_myweb.2d2942f9_myweb-cvd16_default_dc959aad-ba01-11ea-9f79-000c2919d52d_c7802e7f
4 5683c0b919d2 192.168.110.133:5000/pod-infrastructure:latest "/pod" 6 minutes ago Up 6 minutes k8s_POD.37d60359_myweb-cvd16_default_dc959aad-ba01-11ea-9f79-000c2919d52d_919c83e8
5 9b536ac00a54 192.168.110.133:5000/mysql:5.7.30 "docker-entrypoint..." 2 hours ago Up 2 hours k8s_mysql.10e32324_mysql-4j7qk_default_0385e04b-b9ef-11ea-9f79-000c2919d52d_9c9a5b84
6 e4ff14740839 192.168.110.133:5000/pod-infrastructure:latest "/pod" 2 hours ago Up 2 hours k8s_POD.30310355_mysql-4j7qk_default_0385e04b-b9ef-11ea-9f79-000c2919d52d_3a453e5f
7 d2f4e384b94c 192.168.110.133:5000/kubernetes-dashboard-amd64:v1.5.0 "/dashboard --port..." 9 hours ago Up 9 hours k8s_kubernetes-dashboard.4dfe9ff5_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_d5314b66
8 eac876e090b7 192.168.110.133:5000/pod-infrastructure:latest "/pod" 9 hours ago Up 9 hours k8s_POD.3ba4035b_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_c44ca4ec
9 77da48b3cd89 registry "/entrypoint.sh /e..." 11 days ago Up 9 hours 0.0.0.0:5000->5000/tcp registry
10 [root@k8s-node3 ~]# docker exec -it 1caf37bb344e /bin/bash
11 root@myweb-cvd16:/usr/local/tomcat# ls
12 BUILDING.txt CONTRIBUTING.md LICENSE NOTICE README.md RELEASE-NOTES RUNNING.txt bin conf lib logs native-jni-lib temp webapps webapps.dist work
13 root@myweb-cvd16:/usr/local/tomcat# cd webapps
14 root@myweb-cvd16:/usr/local/tomcat/webapps# ls
15 book-master book-master.war
16 root@myweb-cvd16:/usr/local/tomcat/webapps# cd ..
17 root@myweb-cvd16:/usr/local/tomcat# ls
18 BUILDING.txt CONTRIBUTING.md LICENSE NOTICE README.md RELEASE-NOTES RUNNING.txt bin conf lib logs native-jni-lib temp webapps webapps.dist work
19 root@myweb-cvd16:/usr/local/tomcat# cd webapps.dist/
20 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# ls
21 ROOT docs examples host-manager manager
22 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# cp -r * ../webapps
23 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# ls
24 ROOT docs examples host-manager manager
25 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# cd ..
26 root@myweb-cvd16:/usr/local/tomcat# cd webapps
27 root@myweb-cvd16:/usr/local/tomcat/webapps# ls
28 ROOT book-master book-master.war docs examples host-manager manager
29 root@myweb-cvd16:/usr/local/tomcat/webapps#
访问地址http://192.168.110.133:30008/,如下所示:
访问地址http://192.168.110.133:30008/book-master/,如下所示:
需要注意的是,我的项目连接的mysql的地址就是Master节点的ip地址的,然后Pod服务与服务之间的访问,比如tomcat访问mysql的是tomcat连的是mysql的svc的地址,如下所示:
1 drivername=com.mysql.jdbc.Driver
2 url=jdbc:mysql://192.168.110.133:3306/book?useUnicode=true&characterEncoding=utf8
3 user=root
4 password=123456
运行项目可以正常运行,简单的效果,如下所示:
查看kubernetes dashboard的界面如下所示:
点击可以查看详细的日志信息,如下所示:
可以通过命令对Pod的副本数进行调整,如下所示:
1 [root@k8s-master ~]# kubectl scale rc myweb --replicas=2
2 replicationcontroller "myweb" scaled
3 [root@k8s-master ~]# kubectl get all -o wide
4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
5 rc/mysql 1 1 1 3h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
6 rc/myweb 2 2 1 59m myweb 192.168.110.133:5000/tomcat-book:latest app=myweb
7
8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
9 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
10 svc/mysql 10.254.234.52 <none> 3306/TCP 3h app=mysql
11 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 59m app=myweb
12
13 NAME READY STATUS RESTARTS AGE IP NODE
14 po/mysql-4j7qk 1/1 Running 0 3h 172.16.16.3 k8s-node3
15 po/myweb-cvd16 1/1 Running 0 59m 172.16.16.5 k8s-node3
16 po/myweb-sk8qp 0/1 ContainerCreating 0 2s <none> k8s-node2
17 [root@k8s-master ~]# kubectl get all -o wide
18 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
19 rc/mysql 1 1 1 3h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
20 rc/myweb 2 2 2 1h myweb 192.168.110.133:5000/tomcat-book:latest app=myweb
21
22 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
23 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none>
24 svc/mysql 10.254.234.52 <none> 3306/TCP 3h app=mysql
25 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 59m app=myweb
26
27 NAME READY STATUS RESTARTS AGE IP NODE
28 po/mysql-4j7qk 1/1 Running 0 3h 172.16.16.3 k8s-node3
29 po/myweb-cvd16 1/1 Running 0 1h 172.16.16.5 k8s-node3
30 po/myweb-sk8qp 1/1 Running 0 59s 172.16.94.5 k8s-node2
31 [root@k8s-master ~]#
创建的时间大概几十秒吧,我的私有仓库包含项目的tomcat有七八百兆左右,也可以通过kubernetes dashboard进行查看,如下所示:
后来测试如何保证删除了mysql的pod如何保证数据不丢失的时候,发现一个问题的,我发现不管我怎么删除mysql的Pod之后,rc会立马启动一个mysql,但是我的数据就是没有丢失呢,我此时还没有做PV、PVC呢,后来发现原来我用的是192.168.110.133机器安装的mysql,也是服气子了,所以重新改了配置文件,然后重新创建mysql、myweb。下面开始创建mysql,如下所示:
1 [root@k8s-master ~]# cd k8s/
2 [root@k8s-master k8s]# ls
3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume
4 [root@k8s-master k8s]# cd tomcat_demo/
5 [root@k8s-master tomcat_demo]# ls
6 book-master2.0.war book-master3.0.war book-master.war Dockerfile mysql-rc.yml mysql-svc.yml tomcat-rc.yml tomcat-svc.yml
7 [root@k8s-master tomcat_demo]# kubectl create -f mysql-rc.yml
8 replicationcontroller "mysql" created
9 [root@k8s-master tomcat_demo]# kubectl create -f mysql-svc.yml
10 service "mysql" created
11 [root@k8s-master tomcat_demo]# kubectl get all -o wide
12 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
13 rc/mysql 1 1 1 22s mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
14
15 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
16 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
17 svc/mysql 10.254.133.121 <none> 3306/TCP 17s app=mysql
18
19 NAME READY STATUS RESTARTS AGE IP NODE
20 po/mysql-2jtr6 1/1 Running 0 22s 172.16.16.3 k8s-node3
21 [root@k8s-master tomcat_demo]#
创建完毕Mysql之后,如果通过k8s进入到mysql呢,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl exec -it mysql-2jtr6 bash
2 root@mysql-2jtr6:/# mysql -uroot -p123456
3 mysql: [Warning] Using a password on the command line interface can be insecure.
4 Welcome to the MySQL monitor. Commands end with ; or \g.
5 Your MySQL connection id is 2
6 Server version: 5.7.30 MySQL Community Server (GPL)
7
8 Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
9
10 Oracle is a registered trademark of Oracle Corporation and/or its
11 affiliates. Other names may be trademarks of their respective
12 owners.
13
14 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
15
16 mysql> show databases;
17 +--------------------+
18 | Database |
19 +--------------------+
20 | information_schema |
21 | mysql |
22 | performance_schema |
23 | sys |
24 +--------------------+
25 4 rows in set (0.00 sec)
26
27 mysql> CREATE database book DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
28 Query OK, 1 row affected (0.00 sec)
29
30 mysql>
同样,也可以通过访问Pod的ip地址来访问Mysql服务的,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 12h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 11h myweb 192.168.110.133:5000/tomcat-book:4.0 app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
8 svc/mysql 10.254.133.121 <none> 3306/TCP 12h app=mysql
9 svc/myweb 10.254.45.162 <nodes> 8080:30008/TCP 11h app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-2jtr6 1/1 Running 0 12h 172.16.16.3 k8s-node3
13 po/myweb-9qhk9 1/1 Running 0 11h 172.16.16.5 k8s-node3
14 [root@k8s-master tomcat_demo]# mysql -uroot -p123456 -h172.16.16.3
15 mysql: [Warning] Using a password on the command line interface can be insecure.
16 Welcome to the MySQL monitor. Commands end with ; or \g.
17 Your MySQL connection id is 29
18 Server version: 5.7.30 MySQL Community Server (GPL)
19
20 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
21
22 Oracle is a registered trademark of Oracle Corporation and/or its
23 affiliates. Other names may be trademarks of their respective
24 owners.
25
26 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
27
28 mysql>
然后可以创建自己的数据表,由于执行创建数据表的sql过长,这里进行了省略哦,如下所示:
1 mysql> show databases;
2 +--------------------+
3 | Database |
4 +--------------------+
5 | information_schema |
6 | book |
7 | mysql |
8 | performance_schema |
9 | sys |
10 +--------------------+
11 5 rows in set (0.00 sec)
12
13 mysql> use book;
14 Database changed
15 mysql> show tables;
16 Empty set (0.00 sec)
17
18 mysql>
创建好数据表进行查看,如下所示:
1 mysql> show tables;
2 +---------------------+
3 | Tables_in_book |
4 +---------------------+
5 | book_info |
6 | book_recommendation |
7 | book_sort |
8 | user_book |
9 | user_info |
10 +---------------------+
11 5 rows in set (0.00 sec)
12
13 mysql>
然后创建已经将2.x版本的项目打到tomcat的rc、svc,如下所示:
注意:其实这里为了搞清楚java项目的mysql的ip地址、端口号这些填啥,我已经打了很多版本了,最新的是4.0版本的了,操作几乎一样,但是注意下面的配置。
1 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-rc.yml
2 replicationcontroller "myweb" created
3 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-svc.yml
4 service "myweb" created
5 [root@k8s-master tomcat_demo]# kubectl get all -o wide
6 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
7 rc/mysql 1 1 1 6m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
8 rc/myweb 1 1 1 12s myweb 192.168.110.133:5000/tomcat-book:2.0 app=myweb
9
10 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
11 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
12 svc/mysql 10.254.133.121 <none> 3306/TCP 6m app=mysql
13 svc/myweb 10.254.117.144 <nodes> 8080:30008/TCP 8s app=myweb
14
15 NAME READY STATUS RESTARTS AGE IP NODE
16 po/mysql-2jtr6 1/1 Running 0 6m 172.16.16.3 k8s-node3
17 po/myweb-965n1 1/1 Running 0 12s 172.16.16.5 k8s-node3
18 [root@k8s-master tomcat_demo]#
然后开始访问地址http://192.168.110.133:30008/book-master2.0/,k8s访问tomcat报错,源服务器未能找到目标资源的表示或者是不愿公开一个已经存在的资源表示。还可以按照上面的方法进行解决,这里再贴下步骤了,如下所示:
1 [root@k8s-node3 ~]# docker ps
2 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3 a242e75ee4fa 192.168.110.133:5000/tomcat-book:2.0 "catalina.sh run" 3 minutes ago Up 3 minutes k8s_myweb.fc01434c_myweb-965n1_default_b63cbb65-ba10-11ea-9f79-000c2919d52d_dec12b6a
4 e67764bb9b4b 192.168.110.133:5000/pod-infrastructure:latest "/pod" 3 minutes ago Up 3 minutes k8s_POD.37d60359_myweb-965n1_default_b63cbb65-ba10-11ea-9f79-000c2919d52d_57e633de
5 3220e5f8475c 192.168.110.133:5000/mysql:5.7.30 "docker-entrypoint..." 10 minutes ago Up 10 minutes k8s_mysql.10e32324_mysql-2jtr6_default_cbcf1d80-ba0f-11ea-9f79-000c2919d52d_99438f65
6 0d4e3411d100 192.168.110.133:5000/pod-infrastructure:latest "/pod" 10 minutes ago Up 10 minutes k8s_POD.30310355_mysql-2jtr6_default_cbcf1d80-ba0f-11ea-9f79-000c2919d52d_9504bf1f
7 d2f4e384b94c 192.168.110.133:5000/kubernetes-dashboard-amd64:v1.5.0 "/dashboard --port..." 11 hours ago Up 11 hours k8s_kubernetes-dashboard.4dfe9ff5_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_d5314b66
8 eac876e090b7 192.168.110.133:5000/pod-infrastructure:latest "/pod" 11 hours ago Up 11 hours k8s_POD.3ba4035b_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_c44ca4ec
9 77da48b3cd89 registry "/entrypoint.sh /e..." 12 days ago Up 11 hours 0.0.0.0:5000->5000/tcp registry
10 [root@k8s-node3 ~]# docker exec -it a242e75ee4fa /bin/bash
11 root@myweb-965n1:/usr/local/tomcat# cd webapps.dist/
12 root@myweb-965n1:/usr/local/tomcat/webapps.dist# cp -r * ../webapps
13 root@myweb-965n1:/usr/local/tomcat/webapps.dist# ls
14 ROOT docs examples host-manager manager
15 root@myweb-965n1:/usr/local/tomcat/webapps.dist# cd ../webapps
16 root@myweb-965n1:/usr/local/tomcat/webapps#
注意,我这里的java项目使用的就是svc/mysql的ip地址10.254.133.12,然后tomcat-rc.yml由于配置了dns,直接使用的是mysql这个svc的名称。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 48m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 5m myweb 192.168.110.133:5000/tomcat-book:4.0 app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
8 svc/mysql 10.254.133.121 <none> 3306/TCP 48m app=mysql
9 svc/myweb 10.254.45.162 <nodes> 8080:30008/TCP 5m app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-2jtr6 1/1 Running 0 48m 172.16.16.3 k8s-node3
13 po/myweb-9qhk9 1/1 Running 0 5m 172.16.16.5 k8s-node3
14 [root@k8s-master tomcat_demo]#
注意tomcat-rc.yml里面的mysql的ip地址直接使用了mysql的svc名称,如果没有配置dns就需要每次修改mysql的svc的ip地址。
1 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml
2 apiVersion: v1
3 kind: ReplicationController
4 metadata:
5 name: myweb
6 spec:
7 replicas: 1
8 selector:
9 app: myweb
10 template:
11 metadata:
12 labels:
13 app: myweb
14 spec:
15 containers:
16 - name: myweb
17 # image: 192.168.110.133:5000/tomcat:latest
18 image: 192.168.110.133:5000/tomcat-book:4.0
19 ports:
20 - containerPort: 8080
21 env:
22 - name: MYSQL_SERVICE_HOST
23 # value: '192.168.110.133'
24 # value: '10.254.234.52'
25 value: 'mysql'
26 - name: MYSQL_SERVICE_PORT
27 value: '3306'
28 [root@k8s-master tomcat_demo]#
我的java项目的连接mysql数据库的ip地址和端口号直接使用的mysql的svc的ip地址,端口号,这样的缺点就是每次创建mysql都会随机分配一个新的ip地址,就很麻烦的。
1 drivername=com.mysql.jdbc.Driver
2 url=jdbc:mysql://10.254.133.121:3306/book?useUnicode=true&characterEncoding=utf8
3 user=root
4 password=123456
如果按照上面的链接k8s创建的mysql,如果mysql的svc即service重新建一个,会导致这个java连不上mysql数据库的,我又使用了下面这个配置连接k8s里面的mysql数据库,这里使用了svc(service)的名称来代替,如果配置好了dns的话,是可以这样配置的,我的tomcat-rc.yml连接k8s创建的mysql就是这样配置的,如下所示:
1 drivername=com.mysql.jdbc.Driver
2 url=jdbc:mysql://mysql:3306/book?useUnicode=true&characterEncoding=utf8
3 user=root
4 password=123456
此时我的tomcat-rc.yml是这个样子的。
1 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml
2 apiVersion: v1
3 kind: ReplicationController
4 metadata:
5 name: myweb
6 spec:
7 replicas: 1
8 selector:
9 app: myweb
10 template:
11 metadata:
12 labels:
13 app: myweb
14 spec:
15 containers:
16 - name: myweb
17 # image: 192.168.110.133:5000/tomcat:latest
18 image: 192.168.110.133:5000/tomcat-book:5.0
19 ports:
20 - containerPort: 8080
21 env:
22 - name: MYSQL_SERVICE_HOST
23 # value: '192.168.110.133'
24 # value: '10.254.234.52'
25 value: 'mysql'
26 - name: MYSQL_SERVICE_PORT
27 value: '3306'
28 [root@k8s-master tomcat_demo]#
此时,删除mysql的pod验证一下,自己java web如此配置连接k8s里面的Mysql是否正常,然后验证数据没有持久化会造成数据丢失吗?
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 12h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 9m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
8 svc/mysql 10.254.133.121 <none> 3306/TCP 12h app=mysql
9 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 9m app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-2jtr6 1/1 Running 0 12h 172.16.16.3 k8s-node3
13 po/myweb-k23p0 1/1 Running 0 9m 172.16.16.5 k8s-node3
14 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-2jtr6
15 pod "mysql-2jtr6" deleted
16 [root@k8s-master tomcat_demo]# kubectl get all -o wide
17 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
18 rc/mysql 1 1 1 13h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
19 rc/myweb 1 1 1 15m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
20
21 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
22 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
23 svc/mysql 10.254.133.121 <none> 3306/TCP 13h app=mysql
24 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 14m app=myweb
25
26 NAME READY STATUS RESTARTS AGE IP NODE
27 po/mysql-2jtr6 1/1 Terminating 0 13h 172.16.16.3 k8s-node3
28 po/mysql-kt2v2 1/1 Running 0 2s 172.16.16.6 k8s-node3
29 po/myweb-k23p0 1/1 Running 0 15m 172.16.16.5 k8s-node3
30 [root@k8s-master tomcat_demo]#
最终会发现你删除的那个mysql的pod被删除之前,新的mysql的pod已经创建成功了的。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 13h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 16m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
8 svc/mysql 10.254.133.121 <none> 3306/TCP 13h app=mysql
9 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 15m app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-kt2v2 1/1 Running 0 1m 172.16.16.6 k8s-node3
13 po/myweb-k23p0 1/1 Running 0 16m 172.16.16.5 k8s-node3
14 [root@k8s-master tomcat_demo]#
然后登录mysql,这里通过mysql的pod的ip地址进行登录,发现数据已经丢失了,如下所示:
1 [root@k8s-master tomcat_demo]# mysql -uroot -p123456 -h172.16.16.6
2 mysql: [Warning] Using a password on the command line interface can be insecure.
3 Welcome to the MySQL monitor. Commands end with ; or \g.
4 Your MySQL connection id is 4
5 Server version: 5.7.30 MySQL Community Server (GPL)
6
7 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
8
9 Oracle is a registered trademark of Oracle Corporation and/or its
10 affiliates. Other names may be trademarks of their respective
11 owners.
12
13 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
14
15 mysql> show databases;
16 +--------------------+
17 | Database |
18 +--------------------+
19 | information_schema |
20 | mysql |
21 | performance_schema |
22 | sys |
23 +--------------------+
24 4 rows in set (0.01 sec)
25
26 mysql>
或者可以使用k8s的命令进入到Mysql,进行登录,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl exec -it mysql-kt2v2 bash
2 root@mysql-kt2v2:/# mysql -uroot -p123456
3 mysql: [Warning] Using a password on the command line interface can be insecure.
4 Welcome to the MySQL monitor. Commands end with ; or \g.
5 Your MySQL connection id is 5
6 Server version: 5.7.30 MySQL Community Server (GPL)
7
8 Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
9
10 Oracle is a registered trademark of Oracle Corporation and/or its
11 affiliates. Other names may be trademarks of their respective
12 owners.
13
14 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
15
16 mysql> show databases;
17 +--------------------+
18 | Database |
19 +--------------------+
20 | information_schema |
21 | mysql |
22 | performance_schema |
23 | sys |
24 +--------------------+
25 4 rows in set (0.00 sec)
26
27 mysql>
注意:如果Mysql的mysql-svc.yml,这个svc配置了NodePort方式,对外暴漏了端口号,那么在服务器可以通过访问暴漏的端口号,和服务器的ip地址,已经mysql的账号,密码就可以访问k8s里面创建的mysql了的。总之,访问方式有很多种,自己理清楚,自己是通过pod的ip地址(mysql -uroot -p123456 -h172.16.16.6),还是通过k8s的命令访问mysql(kubectl exec -it mysql-kt2v2 bash,然后使用mysql -uroot -p123456查看数据库信息),还是通过svc对外暴漏端口号的方式(这种方式需要配置mysql的svc的yaml配置文件,我这里没有对外暴漏端口号,自己需要的话,可以对外进行暴漏的)。
5、如果上面的Java Web项目没有做持久化之前,如果mysql的pod删除掉,由于配置了rc,k8s会里面启动一个mysql,但是会发现新增的数据没有了,那么如何使用PV、PVC保证数据不丢失呢?这里只是做了mysql的持久化操作。
1 [root@k8s-master ~]# cd k8s/
2 [root@k8s-master k8s]# ls
3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume
4 [root@k8s-master k8s]# cd tomcat_demo/
5 [root@k8s-master tomcat_demo]# ls
6 book-master2.0.war book-master3.0.war book-master4.0.war book-master5.0.war book-master6.0.war book-master.war Dockerfile mysql-rc.yml mysql-svc.yml tomcat-rc.yml tomcat-svc.yml
7 [root@k8s-master tomcat_demo]# cp ../volume/* .
8 [root@k8s-master tomcat_demo]# ls
9 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-svc.yml test-pv.yaml tomcat-svc.yml
10 book-master3.0.war book-master5.0.war book-master.war mysql-rc.yml test-pvc.yaml tomcat-rc.yml
11 [root@k8s-master tomcat_demo]# mv test-pv.yaml mysql-pv.yaml
12 [root@k8s-master tomcat_demo]# mv test-pvc.yaml mysql-pvc.yaml
13 [root@k8s-master tomcat_demo]# ls
14 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-pv.yaml mysql-svc.yml tomcat-svc.yml
15 book-master3.0.war book-master5.0.war book-master.war mysql-pvc.yaml mysql-rc.yml tomcat-rc.yml
16 [root@k8s-master tomcat_demo]#
编辑mysql-pv.yaml、mysql-pvc.yaml两个配置文件,如下所示:
1 apiVersion: v1
2 kind: PersistentVolume
3 metadata:
4 name: mysql
5 labels:
6 type: mysql
7 spec:
8 capacity:
9 storage: 5Gi # 实际生产环境,可以搞大一点的。
10 accessModes:
11 - ReadWriteMany
12 persistentVolumeReclaimPolicy: Recycle
13 nfs:
14 path: "/data/mysql" #拿出一个目录给Mysql来使用。
15 server: 192.168.110.133
16 readOnly: false
将全部的test替换为mysql的用法,如下所示:
然后创建/data/mysql的目录,如下所示:
1 [root@k8s-master tomcat_demo]# mkdir /data/mysql
2 [root@k8s-master tomcat_demo]# ls /data/mysql/
然后修改mysql-pvc.yaml配置文件,如下所示:
1 kind: PersistentVolumeClaim
2 apiVersion: v1
3 metadata:
4 name: mysql
5 spec:
6 accessModes:
7 - ReadWriteMany
8 resources:
9 requests:
10 storage: 5Gi
然后依次创建mysq的PV(全局的资源,被绑定到了default/mysqlpvc这个pvc了)、PVC,并进行查看,如下所示:
1 [root@k8s-master tomcat_demo]# ls
2 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-pv.yaml mysql-rc.yml tomcat-rc.yml
3 book-master3.0.war book-master5.0.war book-master.war mysql-pvc.yaml mysql-rc-pvc.yml mysql-svc.yml tomcat-svc.yml
4 [root@k8s-master tomcat_demo]# kubectl create -f mysql-pv.yaml
5 persistentvolume "mysql" created
6 [root@k8s-master tomcat_demo]# kubectl create -f mysql-pvc.yaml
7 persistentvolumeclaim "mysql" created
8 [root@k8s-master tomcat_demo]# kubectl get pvc
9 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
10 mysql Bound mysql 5Gi RWX 6s
11 [root@k8s-master tomcat_demo]# kubectl get pv
12 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
13 mysql 5Gi RWX Recycle Bound default/mysql 15s
14 [root@k8s-master tomcat_demo]#
那么,现在配置mysql的rc,让它使用卷pvc,这样就可以完成了数据持久化存储了。
1 [root@k8s-master tomcat_demo]# cp mysql-rc.yml mysql-rc-pvc.yml
2 [root@k8s-master tomcat_demo]# vim mysql-rc-pvc.yml
使用的方式就是先增加一个目录,将这个目录挂载出来,比如将/var/lib/mysql目录挂载出来,Pod里面可以有多个容器,所以卷的名字有时候需要写多个,写完多个之后,在使用卷的时候再来声明多个,即volumes的名字必须和volumeMounts挂载的名字是一样的。
具体内容,如下所示:
1 apiVersion: v1
2 kind: ReplicationController
3 metadata:
4 name: mysql
5 spec:
6 replicas: 1
7 selector:
8 app: mysql
9 template:
10 metadata:
11 labels:
12 app: mysql
13 spec:
14 containers:
15 - name: mysql
16 image: 192.168.110.133:5000/mysql:5.7.30
17 ports:
18 - containerPort: 3306
19 env:
20 - name: MYSQL_ROOT_PASSWORD
21 value: '123456'
22 volumeMounts:
23 - name: data
24 mountPath: /var/lib/mysql
25 volumes:
26 - name: data
27 persistentVolumeClaim:
28 claimName: mysql
此时,将mysql-rc-pvc.yml应用一下子,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl apply -f mysql-rc-pvc.yml
2 replicationcontroller "mysql" configured
3 [root@k8s-master tomcat_demo]# kubectl get all -o wide
4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
5 rc/mysql 1 1 1 35m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
6 rc/myweb 1 1 1 35m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
7
8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
9 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
10 svc/mysql 10.254.182.125 <none> 3306/TCP 35m app=mysql
11 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 35m app=myweb
12
13 NAME READY STATUS RESTARTS AGE IP NODE
14 po/mysql-nt1zl 1/1 Running 1 35m 172.16.66.3 k8s-node3
15 po/myweb-c8sf6 1/1 Running 1 35m 172.16.66.5 k8s-node3
16 [root@k8s-master tomcat_demo]#
此时,虽然应用成功了,但是并没有启动一个新的mysql的pod,此时,需要将mysql的pod删除掉,让rc再创建一个新的mysql的pod,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-nt1zl
嗯,很显然,我的这里出现问题了,删除之后,新的mysql的pod启动不起来了,然后我把mysql的rc、svc都删除了,启动新的mysql的rc、svc还是启动不起来了,这里查看一下具体错误信息,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl describe pod mysql-p3mcj
2 Name: mysql-p3mcj
3 Namespace: default
4 Node: k8s-node3/192.168.110.135
5 Start Time: Tue, 30 Jun 2020 12:32:35 +0800
6 Labels: app=mysql
7 Status: Pending
8 IP:
9 Controllers: ReplicationController/mysql
10 Containers:
11 mysql:
12 Container ID:
13 Image: 192.168.110.133:5000/mysql:5.7.30
14 Image ID:
15 Port: 3306/TCP
16 State: Waiting
17 Reason: ContainerCreating
18 Ready: False
19 Restart Count: 0
20 Volume Mounts:
21 /var/lib/mysql from data (rw)
22 Environment Variables:
23 MYSQL_ROOT_PASSWORD: 123456
24 Conditions:
25 Type Status
26 Initialized True
27 Ready False
28 PodScheduled True
29 Volumes:
30 data:
31 Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
32 ClaimName: mysql
33 ReadOnly: false
34 QoS Class: BestEffort
35 Tolerations: <none>
36 Events:
37 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
38 --------- -------- ----- ---- ------------- -------- ------ -------
39 2m 2m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-p3mcj to k8s-node3
40 53s 53s 1 {kubelet k8s-node3} Warning FailedMount Unable to mount volumes for pod "mysql-p3mcj_default(b949a085-ba8a-11ea-af28-000c2919d52d)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-p3mcj". list of unattached/unmounted volumes=[data]
41 53s 53s 1 {kubelet k8s-node3} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-p3mcj". list of unattached/unmounted volumes=[data]
42 47s 47s 1 {kubelet k8s-node3} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/b949a085-ba8a-11ea-af28-000c2919d52d-mysql" (spec.Name: "mysql") pod "b949a085-ba8a-11ea-af28-000c2919d52d" (UID: "b949a085-ba8a-11ea-af28-000c2919d52d") with: mount failed: exit status 32
43 Mounting command: mount
44 Mounting arguments: 192.168.110.133:/data/mysql /var/lib/kubelet/pods/b949a085-ba8a-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql nfs []
45 Output: mount.nfs: Connection refused
46
47
48 [root@k8s-master tomcat_demo]#
出现这种挂载不了的问题,就是nfs没有配置好,下面可以再配置一下,如下所示:
关于多个centos7操作系统间实现文件数据资源共享(挂载mount网络文件路径),这边通过在服务端配置nfs(网络文件系统)+rpcbind服务软件,在三台机器上面确定安装了nfs,如下所示:
1 [root@k8s-master ~]# rpm -qa|grep nfs
2 nfs-utils-1.3.0-0.66.el7.x86_64
3 libnfsidmap-0.25-19.el7.x86_64
4 [root@k8s-master ~]#
确认三台服务端都安装配置rpcbind、如下所示:
1 [root@k8s-master ~]# service nfs start
2 Redirecting to /bin/systemctl start nfs.service
在服务端的/etc/exports配置共享路径,并重启服务,同时关闭防火墙(比较重要,否则报mount.nfs: Connection timed out)。
1 [root@k8s-master ~]# cat /etc/exports
2 # 共享data目录,允许192.168.110.*访问,即允许110段ip地址访问。读写权限,同步,不做root用户、所有用户的UID映射
3 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash)
4
5
6 [root@k8s-master ~]#
设置/data目录的权限,如下所示:
1 [root@k8s-master ~]# ll /data/
2 total 0
3 drwxr-xr-x 2 root root 6 Jun 29 17:21 k8s
4 drwxr-xr-x 5 polkitd root 328 Jun 30 13:04 mysql
5 [root@k8s-master ~]# chmod -R 755 /data/
6 [root@k8s-master ~]# ll /data/
7 total 0
8 drwxr-xr-x 2 root root 6 Jun 29 17:21 k8s
9 drwxr-xr-x 5 polkitd root 328 Jun 30 13:04 mysql
10 [root@k8s-master ~]#
在客户端mount上面服务端分享的nfs路径/data(绑定在客户端的/data),测试成功。
1 [root@k8s-node2 ~]# mount -t nfs 192.168.110.133:/data /data
2 mount.nfs: mount point /data does not exist
3 [root@k8s-node2 ~]# cd mkdir /data
4 -bash: cd: mkdir: No such file or directory
5 [root@k8s-node2 ~]# mkdir /data
6 [root@k8s-node2 ~]# mount -t nfs 192.168.110.133:/data /data
7 [root@k8s-node2 ~]# cd /data/
8 [root@k8s-node2 data]# ls
9 k8s mysql
10 [root@k8s-node2 data]#
挂载成功之后,重启两个客户端的nfs服务、rpcbind服务,如下所示:
1 [root@k8s-node2 data]# systemctl restart rpcbind.service
2 [root@k8s-node2 data]# systemctl restart nfs.service
3 [root@k8s-node2 data]#
4
5 [root@k8s-node3 data]# systemctl restart rpcbind.service
6 [root@k8s-node3 data]# systemctl restart nfs.service
7 [root@k8s-node3 data]#
此时,再次查看创建的mysql的Pod是否已经创建成功了,如下所示:
1 [root@k8s-master tomcat_demo]# kubectl describe pod mysql-njrhc
2 Name: mysql-njrhc
3 Namespace: default
4 Node: k8s-node3/192.168.110.135
5 Start Time: Tue, 30 Jun 2020 12:42:45 +0800
6 Labels: app=mysql
7 Status: Running
8 IP: 172.16.66.3
9 Controllers: ReplicationController/mysql
10 Containers:
11 mysql:
12 Container ID: docker://a689aeadd94774508b8f384befb62124e28d208801dda204aa4e36e24c476960
13 Image: 192.168.110.133:5000/mysql:5.7.30
14 Image ID: docker-pullable://192.168.110.133:5000/mysql@sha256:0563b36ec2d1a262f79e1d8562e61f642a0f64f93306d8a709047cdea0444d0a
15 Port: 3306/TCP
16 State: Running
17 Started: Tue, 30 Jun 2020 13:03:43 +0800
18 Ready: True
19 Restart Count: 0
20 Volume Mounts:
21 /var/lib/mysql from data (rw)
22 Environment Variables:
23 MYSQL_ROOT_PASSWORD: 123456
24 Conditions:
25 Type Status
26 Initialized True
27 Ready True
28 PodScheduled True
29 Volumes:
30 data:
31 Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
32 ClaimName: mysql
33 ReadOnly: false
34 QoS Class: BestEffort
35 Tolerations: <none>
36 Events:
37 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
38 --------- -------- ----- ---- ------------- -------- ------ -------
39 27m 27m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-njrhc to k8s-node3
40 25m 9m 8 {kubelet k8s-node3} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/25204e19-ba8c-11ea-af28-000c2919d52d-mysql" (spec.Name: "mysql") pod "25204e19-ba8c-11ea-af28-000c2919d52d" (UID: "25204e19-ba8c-11ea-af28-000c2919d52d") with: mount failed: exit status 32
41 Mounting command: mount
42 Mounting arguments: 192.168.110.133:/data/mysql /var/lib/kubelet/pods/25204e19-ba8c-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql nfs []
43 Output: mount.nfs: Connection refused
44
45
46 25m 7m 9 {kubelet k8s-node3} Warning FailedMount Unable to mount volumes for pod "mysql-njrhc_default(25204e19-ba8c-11ea-af28-000c2919d52d)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-njrhc". list of unattached/unmounted volumes=[data]
47 25m 7m 9 {kubelet k8s-node3} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-njrhc". list of unattached/unmounted volumes=[data]
48 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Pulled Container image "192.168.110.133:5000/mysql:5.7.30" already present on machine
49 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Created Created container with docker id a689aeadd947; Security:[seccomp=unconfined]
50 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Started Started container with docker id a689aeadd947
51 [root@k8s-master tomcat_demo]# kubectl get all -o wide
52 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
53 rc/mysql 1 1 1 38m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
54 rc/myweb 1 1 1 1h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
55
56 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
57 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
58 svc/mysql 10.254.126.11 <none> 3306/TCP 38m app=mysql
59 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 1h app=myweb
60
61 NAME READY STATUS RESTARTS AGE IP NODE
62 po/mysql-njrhc 1/1 Running 0 28m 172.16.66.3 k8s-node3
63 po/myweb-c8sf6 1/1 Running 1 1h 172.16.66.5 k8s-node3
64 [root@k8s-master tomcat_demo]#
由于此时,两个客户端都挂载了服务器端的目录,此时查看三台机器的/data目录,发现里面的内容完全一样,如下所示:
1 [root@k8s-master ~]# cd /data/
2 [root@k8s-master data]# ls
3 k8s mysql
4 [root@k8s-master data]# cd mysql/
5 [root@k8s-master mysql]# ls
6 auto.cnf ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys
7 ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem
8 [root@k8s-master mysql]#
可以查看挂载信息,查看一下mysql的pod是在那个机器上启动的,然后使用命令查看挂载信息。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 46m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
8 svc/mysql 10.254.126.11 <none> 3306/TCP 46m app=mysql
9 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/mysql-njrhc 1/1 Running 0 36m 172.16.66.3 k8s-node3
13 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3
14 [root@k8s-master tomcat_demo]#
可以在Node3节点使用df -h命令查看挂载信息,如下所示:
1 [root@k8s-node3 mysql]# df -h
2 Filesystem Size Used Avail Use% Mounted on
3 /dev/mapper/centos-root 10G 6.5G 3.6G 65% /
4 devtmpfs 1.2G 0 1.2G 0% /dev
5 tmpfs 1.2G 0 1.2G 0% /dev/shm
6 tmpfs 1.2G 11M 1.2G 1% /run
7 tmpfs 1.2G 0 1.2G 0% /sys/fs/cgroup
8 /dev/sda1 197M 157M 41M 80% /boot
9 tmpfs 229M 12K 229M 1% /run/user/42
10 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/5265b3c878e49cfa04607df2b3d650eb4a66d2eff93edbabc9bbccffcb3cb5fb/merged
11 shm 64M 0 64M 0% /var/lib/docker/containers/77da48b3cd89ac31f2e1670978d6c4258bf5a5412b337a639776df4a60bc0264/shm
12 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/ab511d9e7a8f54e2ef11de6e809047bbfde443d1eb3244cbd0f6a8fe32d51552/merged
13 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/dc3decab78b1c5385384af3199727c41e7e05a96f769288cd133b1831e8b8cff/merged
14 shm 64M 0 64M 0% /var/lib/docker/containers/bad9be6adcd79953768767f69cab35515175fa08ac6cb924dadc40e8c52bd672/shm
15 shm 64M 0 64M 0% /var/lib/docker/containers/6700afcaaaa7115c9079a4ebb988dd6293edd6a5e880bf1b8d1bab4f023ae657/shm
16 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/2e1480e2559a602d0a0f5437113829ac5f50d2c09f38000aacdea87be0237616/merged
17 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/98519ce1d9e7d7712cc0aec0482fc0e3e246c9f8dfb9e6cbda8517e97b1a5063/merged
18 tmpfs 229M 0 229M 0% /run/user/0
19 192.168.110.133:/data/mysql 18G 14G 3.9G 79% /var/lib/kubelet/pods/25204e19-ba8c-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql
20 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/9cbf0877f0dc6abef9cfa3aba292066f36f7d7a12f50d642c98be91d62b97c1b/merged
21 shm 64M 0 64M 0% /var/lib/docker/containers/7e03fb4382a25120de8a8714a2c9210dd06b7bb09b94900f963e701d077b5843/shm
22 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/64614fb76966f8d630d77bc333b24fa67c2078f711a05cf958b2e32c6bab36f5/merged
23 192.168.110.133:/data 18G 14G 3.9G 79% /data
24 [root@k8s-node3 mysql]#
然后可以在web界面录入几条新的数据,然后删除mysql的pod,开始验证是否已经做到持久化了,如下所示:
删除mysql的pod,操作如下所示:
1 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-njrhc
2 pod "mysql-njrhc" deleted
3 [root@k8s-master tomcat_demo]# kubectl get all -o wide
4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
5 rc/mysql 1 1 0 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
6 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
7
8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
9 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
10 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql
11 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb
12
13 NAME READY STATUS RESTARTS AGE IP NODE
14 po/mysql-njrhc 1/1 Terminating 0 40m 172.16.66.3 k8s-node3
15 po/mysql-wldks 0/1 ContainerCreating 0 1s <none> k8s-node3
16 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3
17 [root@k8s-master tomcat_demo]# kubectl get all -o wide
18 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
19 rc/mysql 1 1 1 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
20 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
21
22 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
23 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
24 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql
25 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb
26
27 NAME READY STATUS RESTARTS AGE IP NODE
28 po/mysql-njrhc 1/1 Terminating 0 40m 172.16.66.3 k8s-node3
29 po/mysql-wldks 1/1 Running 0 3s 172.16.66.6 k8s-node3
30 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3
31 [root@k8s-master tomcat_demo]# kubectl get all -o wide
32 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
33 rc/mysql 1 1 1 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
34 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb
35
36 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
37 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none>
38 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql
39 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb
40
41 NAME READY STATUS RESTARTS AGE IP NODE
42 po/mysql-wldks 1/1 Running 0 5s 172.16.66.6 k8s-node3
43 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3
44 [root@k8s-master tomcat_demo]#
然后查看界面上的新录入的数据,是否还存在,如下所示:
然后,查看一下挂载的目录也是有内容的,如下所示:
1 [root@k8s-node3 mysql]# ls
2 auto.cnf ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem
3 book ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys
4 [root@k8s-node3 mysql]#