使用Operator在K8S部署MySQL

运维部落

发布于 2022-01-18 10:45:03

5.3K1

发布于 2022-01-18 10:45:03

文章被收录于专栏：运维部落运维部落

使用Operator在K8S部署MySQL

一. 本文概述
- 1.1 技术选型
- 1.2 全文实操内容如下
二. 本文主要术语及工具版本
三. 添加Bitpoke源及安装Operator
四. MySQL单实例安装
- 4.1 自动备份及恢复验证
- 4.2 NFS存储验证
五. MySQL集群安装
- 5.1 高可用及主从复制验证
六. 参考资料

一. 本文概述

目前研究如何在K8S上使用MySQL Operator对MySQL进行部署及管理，汇总研究过程形成此文，分享出来希望能对学习者提供帮助，若有建议请不吝指出。

1.1 技术选型

当前主要有Oracle MySQL Operator和Presslabs MySQL Operator两种方案，前者由Oracle官方发布，使用MGR架构，目前仅支持MySQL 8.0+且缺少维护；后者则由Bitpoke进行维护，支持MySQL 5.7 、提供备份恢复和基于Orchestrator实现高可用等功能，本文基于后者编写。

1.2 全文实操内容如下

MySQL单实例上基于Amazon s3定时进行云端备份，利用备份的数据进行恢复验证备份有效性。
部署MySQL单实例 + NFS存储，当Pod故障且无法被拉起时，在新Node启动的Pod借助NFS可读取原主实例写入的数据。
部署MySQL集群，一主两从架构，验证高可用及主从复制的有效性。

二. 本文主要术语及工具版本

Kubernetes Operator: 是一种封装、部署和管理 Kubernetes 应用的方法。我们使用 Kubernetes API（应用编程接口）和 kubectl 工具在 Kubernetes 上部署并管理 Kubernetes 应用。
Helm：是Kubernetes的包管理器，类似于Centos的yum,主要用来管理 Charts。
Helm Charts: 是用来封装Kubernetes原生应用程序的一系列YAML文件,可在部署应用时自定义应用程序的Metadata。
Helm version : v3.2.4
Kubernetes version: v1.19.12
MySQL version: v5.7.31
系统版本：CentOS Linux release 7.9.2009 (Core)

三. 添加Bitpoke源及安装Operator

此PV用于安装Bitpoke MySQL Operator

[root@65-89-master]# cat pv.yaml 

apiVersion: v1

kind: PersistentVolume

metadata:

 name: pv-normal0

 labels:

  type: local

spec:

 capacity:

  storage: 10Gi

 accessModes:

  - ReadWriteOnce

 hostPath:

path: "/data/pv0"

创建PV

[root@65-89-master]# kubectl apply -f pv.yaml

persistentvolume/pv-normal0 created

确认PV已创建

[root@65-89-master ~]# kubectl get pv

NAME CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS   CLAIM  STORAGECLASS  REASON  AGE

pv-normal0  10Gi    RWO       Retain      Available                                 39h

添加helm repo进行并update

helm repo add bitpoke https://helm-charts.bitpoke.io

helm update

安装mysql-operator，以下根据helm版本二选一，本文为v3

# helm v2

helm install bitpoke/mysql-operator --name mysql-operator --create-namespace

# helm v3 

helm install mysql-operator bitpoke/mysql-operator -n mysql-operator --create-namespace

确认mysql-operator状态为Running

[root@65-89-master ~]# kubectl get pod -n mysql-operator

NAME              READY  STATUS  RESTARTS  AGE

mysql-operator-0   2/2   Running  0        39h

四. MySQL单实例安装

将明文密码进行base64格式编码

[root@65-89-master single]# echo -n 'ZZQzzq123' | base64

WlpRenpxMTIz

将编码后的字符设置为MySQL中root用户的密码

[root@65-89-master single]# cat secret.yaml

apiVersion: v1

kind: Secret

metadata:

name: my-secret

namespace: mysql-operator

type: Opaque

data:

ROOT_PASSWORD: WlpRenpxMTIz

创建MySQL secret

[root@65-89-master single]# kubectl apply -f secret.yaml

secret/my-secret created

查看Amazon s3 secret

[root@65-89-master single]# cat example-backup-secret.yaml 

apiVersion: v1

kind: Secret

metadata:

name: my-cluster-backup-secret

namespace: mysql-operator

type: Opaque

data:

AWS_ACCESS_KEY_ID: QU************************

AWS_SECRET_ACCESS_KEY: ME9B****************

创建Amazon s3 secret

[root@65-89-master single]# kubectl apply -f example-backup-secret.yaml

secret/my-cluster-backup-secret created

确认secret已创建

NAME                         TYPE                                    DATA  AGE

default-token-b5jbh          kubernetes.io/service-account-token     3    4d18h

my-cluster-backup-secret     Opaque                                  2    99m

my-secret                    Opaque                                  1    3d3h

查看PV，使用此前已创建的NFS服务

[root@65-89-master single]# cat pv-nfs.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

name: mysql-nfs0

spec:

capacity:

storage: 10Gi

volumeMode: Filesystem

accessModes:

ReadWriteOnce

persistentVolumeReclaimPolicy: Retain

storageClassName: nfs

nfs:

path: /nfs/mysql

server: 10.186.65.89

创建PV

[root@65-89-master single]# kubectl apply -f pv-nfs.yaml

persistentvolume/mysql-nfs0 created

确认PV就绪

NAME  CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS   CLAIM  STORAGECLASS  REASON  AGE

mysql-nfs0  10Gi    RWO      Retain     Available                 nfs                12s

pv-normal0  10Gi    RWO      Retain     Bound  mysql-operator/data-mysql-operator-0   40h

查看MySQL单实例配置文件

每天15：46自动生成s3备份

[root@65-89-master single]# cat single.yaml

apiVersion: mysql.presslabs.org/v1alpha1

kind: MysqlCluster

metadata:

name: single

namespace: mysql-operator

spec:

mysqlVersion: "5.7"

replicas: 1

secretName: my-secret

backupSchedule: "46 15 * * *"   

backupURL: s3://evan-zheng-***************************  

backupSecretName: my-cluster-backup-secret 

backupRemoteDeletePolicy: retain  

mysqlConf:

max_allowed_packet: "512M" 

volumeSpec:

persistentVolumeClaim:

storageClassName: nfs

accessModes:

ReadWriteOnce

resources:

requests:

storage: 10Gi

创建MySQL单实例

[root@65-89-master single]# kubectl apply -f single.yaml

mysqlcluster.mysql.presslabs.org/single created

查看pod状态为Running

[root@65-89-master single]# kubectl get pod -n mysql-operator

NAME        READY  STATUS  RESTARTS  AGE

mysql-operator-0  2/2   Running  0     39h

single-mysql-0    4/4   Running  0     5m15s

确认MySQL状态为True

[root@65-89-master single]# kubectl get mysql -n mysql-operator

NAME   READY  REPLICAS  AGE

single    True    1     5m35s

进入MySQL 的pod，创建库表并插入数据。

[root@65-89-master single]# kubectl exec -it single-mysql-0 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to mysql.
Use 'kubectl describe pod/single-mysql-0 -n mysql-operator' to see all of the containers in this pod.
bash-4.2$ mysql -uroot -p'ZZQzzq123'

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 485

Server version: 5.7.31-34-log Percona Server (GPL), Release 34, Revision 2e68637

Copyright (c) 2009-2020 Percona LLC and/or its affiliates

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database quan;

Query OK, 1 row affected (0.05 sec)

mysql> use quan

Database changed

mysql> create table test(id int);

Query OK, 0 rows affected (0.15 sec)

mysql> insert into test values(1),(2),(3);

Query OK, 3 rows affected (0.01 sec)

Records: 3 Duplicates: 0 Warnings: 0

mysql> 

mysql> select * from test;

+------+

| id  |

+------+

|  1 |

|  2 |

|  3 |

+------+

3 rows in set (0.00 sec)

4.1 自动备份及恢复验证

确认自动自动备份已生成

[root@65-89-master s3-bak]# kubectl get mysqlbackup -n mysql-operator

NAME                              AGE

single-auto-2021-12-29t08-15-46   28m

登录Amazon s3确认bucket已生成备份文件

复制备份文件S3 URI

基于云端备份恢复实例

[root@65-89-master single]# cat recovery.yaml 

apiVersion: mysql.presslabs.org/v1alpha1

kind: MysqlCluster

metadata:

 name: rec-cluster

 namespace: mysql-operator

spec:

 secretName: my-secret

 initBucketURL: s3://evan-zheng-bucket/single-auto-2021-12-29t08-15-46.xbackup.gz

 initBucketSecretName: my-cluster-backup-secret

创建“恢复”实例

[root@65-89-master s3-bak]# kubectl apply -f recovery.yaml

mysqlbackup.mysql.presslabs.org/rec-cluster created

确认实例状态为Running

[root@65-89-master single]# kubectl get pod -n mysql-operator

NAME                               READY  STATUS   RESTARTS  AGE

rec-cluster-mysql-0                4/4    Running   0        2m16s

验证恢复数据

[root@65-89-master single]#  kubectl exec -it rec-cluster-mysql-0 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Defaulting container name to mysql.

Use 'kubectl describe pod/rec-cluster-mysql-0 -n mysql-operator' to see all of the containers in this pod.

bash-4.2$ 

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'use quan; select * from test;'

mysql: [Warning] Using a password on the command line interface can be insecure.

+------+

| id  |

+------+

|  1 |

|  2 |

|  3 |

+------+

4.2 NFS存储验证

查看当前pod所在node为65-37

[root@65-89-master single]# kubectl describe pod single-mysql-0 -n mysql-operator | grep node

Node:     65-37-node2/10.186.65.37

使65-37不参与调度

[root@65-89-master single]# kubectl cordon 65-37-node2

node/65-37-node2 cordoned

确认65-37不参与调度

[root@65-89-master single]# kubectl get node

NAME          STATUS                    ROLES           AGE     VERSION

65-37-node2   Ready,SchedulingDisabled  worker           2d17h  v1.19.16

65-70-node1   Ready                     worker           2d17h  v1.19.16

65-89-master  Ready           controlplane,etcd,worker   2d17h  v1.19.16

删除MySQL当前pod

[root@65-89-master single]# kubectl delete pod single-mysql-0 -n mysql-operator

pod "single-mysql-0" deleted

查看pod已自动重新初始化

[root@65-89-master single]# kubectl get pod -n mysql-operator

NAME             READY  STATUS           RESTARTS    AGE

mysql-operator-0  2/2   Running           0          40h

single-mysql-0    0/4   PodInitializing   0          16s

查看当前pod所在node已变更为65-70

[root@65-89-master single]# kubectl describe pod single-mysql-0 -n mysql-operator | grep node

Node:     65-70-node1/10.186.65.70

进入pod并确认此前插入的数据仍旧存在

[root@65-89-master single]# kubectl exec -it single-mysql-0 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Defaulting container name to mysql.

Use 'kubectl describe pod/single-mysql-0 -n mysql-operator' to see all of the containers in this pod.

bash-4.2$ 

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'select * from quan.test;'

mysql: [Warning] Using a password on the command line interface can be insecure.

+------+

| id  |

+------+

|  1 |

|  2 |

|  3 |

+------+

使65-37参与调度

[root@65-89-master single]# kubectl uncordon 65-37-node2

node/65-37-node2 uncordoned

确认65-37可参与调度

[root@65-89-master single]# kubectl get node

NAME          STATUS   ROLES                   AGE      VERSION

65-37-node2   Ready    worker                  2d17h    v1.19.16

65-70-node1   Ready    worker                  2d17h    v1.19.16

65-89-master  Ready  controlplane,etcd,worker  2d17h    v1.19.16

五. MySQL集群安装

查看MySQL集群PV文件

[root@65-89-master cluster]# cat pv-manu.yaml 

apiVersion: v1

kind: PersistentVolume

metadata:

 name: pv-mysql1

 labels:

  type: local

spec:

 storageClassName: manual

 capacity:

  storage: 10Gi

 accessModes:

  - ReadWriteOnce

 hostPath:

  path: "/data/mydata"

 
---

apiVersion: v1

kind: PersistentVolume

metadata:

 name: pv-mysql2

 labels:

  type: local

spec:

 storageClassName: manual

 capacity:

  storage: 10Gi

 accessModes:

  - ReadWriteOnce

 hostPath:

  path: "/data/mydata"
 

---

apiVersion: v1

kind: PersistentVolume

metadata:

 name: pv-mysql3

 labels:

  type: local

spec:

 storageClassName: manual

 capacity:

  storage: 10Gi

 accessModes:

  - ReadWriteOnce

 hostPath:

path: "/data/mydata"

创建及确认PV状态不再赘述
查看MySQL集群配置文件

[root@65-89-master cluster]# cat clu.yaml 

apiVersion: mysql.presslabs.org/v1alpha1

kind: MysqlCluster

metadata:

 name: mysql-cluster

 namespace: mysql-operator

spec:

 mysqlVersion: "5.7"

 replicas: 3

 secretName: my-secret

 mysqlConf:

  max_allowed_packet: "512M" 

 volumeSpec:

  persistentVolumeClaim:

   storageClassName: manual

   accessModes:

   - ReadWriteOnce

   resources:

     requests:

      storage: 10Gi

创建MySQL集群

[root@65-89-master cluster]# kubectl apply -f clu.yaml

mysqlcluster.mysql.presslabs.org/mysql-cluster created

确认pod均已Running

[root@65-89-master cluster]# kubectl get pod -n mysql-operator

NAME                     READY  STATUS   RESTARTS  AGE

mysql-cluster-mysql-0    4/4    Running   0        17m

mysql-cluster-mysql-1    4/4    Running   0        36m

mysql-cluster-mysql-2    4/4    Running   0        35m

mysql-operator-0         2/2    Running   0        40h

进入pod mysql-cluster-mysql-0确认其为Master

root@65-89-master cluster]# kubectl exec -it mysql-cluster-mysql-0 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Defaulting container name to mysql.

Use 'kubectl describe pod/mysql-cluster-mysql-0 -n mysql-operator' to see all of the containers in this pod.

bash-4.2$

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'show slave status\G'

mysql: [Warning] Using a password on the command line interface can be insecure.

bash-4.2$

5.1 高可用及主从复制验证

删除Master引发主从切换，验证高可用有效性

[root@65-89-master cluster]# kubectl delete pod mysql-cluster-mysql-0 -n mysql-operator | grep mysql

pod "mysql-cluster-mysql-0" deleted

此时mysql-cluster-mysql-1已成为Master，在其创建库表并插入数据。

[root@65-89-master ~]# kubectl exec -it mysql-cluster-mysql-1 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Defaulting container name to mysql.

Use 'kubectl describe pod/mysql-cluster-mysql-1 -n mysql-operator' to see all of the containers in this pod.

bash-4.2$  

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'show slave status\G'

mysql: [Warning] Using a password on the command line interface can be insecure.

bash-4.2$ 

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'create database cluster; use cluster; create table demo(id int); insert into demo values(4),(5),(6);select * from demo;'

mysql: [Warning] Using a password on the command line interface can be insecure.

+------+

| id  |

+------+

|  4 |

|  5 |

|  6 |

+------+

bash-4.2$

进入Slave mysql-cluster-mysql-2查询主库创建的数据

[root@65-89-master ~]# kubectl exec -it mysql-cluster-mysql-2 -n mysql-operator /bin/bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Defaulting container name to mysql.

Use 'kubectl describe pod/mysql-cluster-mysql-2 -n mysql-operator' to see all of the containers in this pod.

bash-4.2$

bash-4.2$ mysql -uroot -p'ZZQzzq123' -e 'select * from cluster.demo;'

mysql: [Warning] Using a password on the command line interface can be insecure.

+------+

| id  |

+------+

|  4 |

|  5 |

|  6 |

+------+