OpenEBS Local Volume 测试
OpenEBS Local Volume 测试
OpenEBS Local Volume
Local PV hostpath
对比Kubernetes Hostpath卷相比，OpenEBS本地PV Hostpath卷具有以下优势:
OpenEBS本地PV Hostpath允许您的应用程序通过StorageClass、PVC和PV访问Hostpath。 这为您提供了更改PV提供者的灵活性，而无需重新设计应用程序YAML- 使用
Velero备份和恢复进行数据保护 - 通过对应用程序
YAML和pod完全屏蔽主机路径来防范主机路径安全漏洞
挂载分区
创建 1T 的 LVM 挂载到 /var/openebs/local 目录（local pv hostpath 默认使用目录，也可以创建新的 StorageClass 指定新的目录）
1 2 | lvcreate -n lv_openebs -L 1T datavg ;mkfs.xfs /dev/datavg/lv_openebs ;mkdir -p /var/openebs/local mount /dev/datavg/lv_openebs /var/openebs/local |
|---|
安装
1 | kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml |
|---|
默认创建两个 storageclass ，我们主要使用 openebs-hostpath
1 2 3 4 | # kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 5m43s openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 5m43s |
|---|
验证
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | # vim local-hostpath-sample.yaml --- #创建 pvc kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-hostpath-pvc spec: storageClassName: openebs-hostpath accessModes: - ReadWriteOnce resources: requests: storage: 5G --- # 创建 pod apiVersion: v1 kind: Pod metadata: name: hello-local-hostpath-pod spec: volumes: - name: local-storage persistentVolumeClaim: claimName: local-hostpath-pvc containers: - name: hello-container image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done' volumeMounts: - mountPath: /mnt/store name: local-storage kubectl apply -f local-hostpath-sample.yaml |
|---|
查看 pv 和 pvc
1 2 3 4 5 6 7 | # kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-a500bbdc-65d8-4b38-a342-c15828e4a358 5G RWO Delete Bound default/local-hostpath-pvc openebs-hostpath 6m50s # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE local-hostpath-pvc Bound pvc-a500bbdc-65d8-4b38-a342-c15828e4a358 5G RWO openebs-hostpath 8m20s |
|---|
对应节点的 /var/openebs/local 目录
1 2 3 4 5 6 7 8 9 | # pod 路径 # kubectl exec -ti hello-local-hostpath-pod -- cat /mnt/store/greet.txt Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. # 主机路径 # cat /var/openebs/local/pvc-a500bbdc-65d8-4b38-a342-c15828e4a358/greet.txt Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. |
|---|
清理 pod
1 | kubectl delete pod hello-local-hostpath-pod |
|---|
性能测试
下载性能测试文件：
performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark
调整以下内容：
1 2 3 4 | volumes: - name: dbench-pv persistentVolumeClaim: claimName: local-hostpath-pvc # 修改这里 |
|---|
创建应用
1 | kubectl create -f fio-deploy.yaml |
|---|
查看状态
1 2 3 | # kubectl get pod NAME READY STATUS RESTARTS AGE dbench-zp7th--1-xjj8t 1/1 Running 0 5s |
|---|
查看性能测试结果
1 2 3 4 5 6 7 8 9 10 11 | kubectl logs dbench-zp7th--1-xjj8t …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 130k/25.3k. BW: 2838MiB/s / 978MiB/s Average Latency (usec) Read/Write: 93.60/384.73 Sequential Read/Write: 4128MiB/s / 1673MiB/s Mixed Random Read/Write IOPS: 25.1k/8413 |
|---|
清理
1 2 | kubectl delete pod dbench-zp7th--1-xjj8t kubectl delete pvc local-hostpath-pvc |
|---|
Local PV device
对比Kubernetes本地持久卷，OpenEBS本地PV设备卷有以下优点:
OpenEBS本地PV设备卷provider是动态的，Kubernetes设备卷provider是静态的OpenEBS NDM更好地管理用于创建本地pv的块设备。NDM提供了发现块设备属性、设置设备筛选器、度量集合以及检测块设备是否已经跨节点移动等功能
创建 SC
默认安装已经自带了 local PV 这里可以跳过
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | cat > local-device-sc.yaml <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-device annotations: openebs.io/cas-type: local cas.openebs.io/config: | - name: StorageType value: device - name: FSType value: xfs provisioner: openebs.io/local reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer EOF kubectl apply -f local-device-sc.yaml |
|---|
验证
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | # vim local-pv-sample.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-device-pvc spec: storageClassName: local-device accessModes: - ReadWriteOnce resources: requests: storage: 5G --- apiVersion: v1 kind: Pod metadata: name: hello-local-device-pod spec: volumes: - name: local-storage persistentVolumeClaim: claimName: local-device-pvc containers: - name: hello-container image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done' volumeMounts: - mountPath: /mnt/store name: local-storage kubectl apply -f local-device-pod.yaml |
|---|
查看 pv 和 pvc
1 2 3 4 5 6 7 | # kubectl get pod NAME READY STATUS RESTARTS AGE hello-local-device-pod 1/1 Running 0 10m # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE local-device-pvc Bound pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce 5G RWO local-device 10m |
|---|
查看磁盘（ndm会自动选择合适的数据盘）
如下所示，sdb 被使用了，自动选择了 sdc 创建分区挂载到 pod
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/bc10974f-1a8a-433f-84b9-b20d8dc908c4/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce |
|---|
性能测试
创建 pvc 用于性能测试
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | cat > dbench-pvc.yaml <<EOF --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: dbench spec: storageClassName: local-device accessModes: - ReadWriteOnce resources: requests: storage: 5G EOF kubectl apply -f dbench-pvc.yaml |
|---|
下载性能测试文件：
performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark
调整以下内容：
1 2 3 4 | volumes: - name: dbench-pv persistentVolumeClaim: claimName: dbench # 修改这里 |
|---|
创建应用
1 | kubectl create -f fio-deploy.yaml |
|---|
查看状态
1 2 3 | # kubectl get pod NAME READY STATUS RESTARTS AGE dbench-qzczk--1-kz6wf 1/1 Running 0 10s |
|---|
查看磁盘挂载信息
自动选择了另外一台机器的 sdc 创建分区挂载到 pod
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | 10.188.100.99 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/0ebab420-cdfa-4421-8a50-3fcdbe59ac94/volumes/kubernetes.io~local-volume/pvc-c2206bd7-dbd9-45d2-ba4b-36224edbd0e4 …… 10.188.100.101 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/e8ba48b4-957d-4d6f-bc5f-d850f058d603/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce |
|---|
查看性能测试结果
性能与 hostpath 模式相差不大
1 2 3 4 5 6 7 8 9 10 11 | kubectl logs dbench-qzczk--1-kz6wf …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 149k/25.7k. BW: 2474MiB/s / 997MiB/s Average Latency (usec) Read/Write: 94.81/384.69 Sequential Read/Write: 4428MiB/s / 1821MiB/s Mixed Random Read/Write IOPS: 22.4k/7434 |
|---|
清理
1 2 | kubectl delete pod dbench-qzczk--1-kz6wf hello-local-device-pod kubectl delete pvc dbench local-device-pvc |
|---|
LVM Local PV
安装
1 | kubectl apply -f https://openebs.github.io/charts/lvm-operator.yaml |
|---|
查看 pod 状态
1 2 3 4 5 6 7 8 9 10 11 12 | # kubectl get pod -A | grep openebs kube-system openebs-lvm-controller-0 5/5 Running 0 74s kube-system openebs-lvm-node-47nz2 2/2 Running 0 74s kube-system openebs-lvm-node-7nl47 2/2 Running 0 74s kube-system openebs-lvm-node-dwrkc 2/2 Running 0 74s kube-system openebs-lvm-node-g55vx 2/2 Running 0 74s kube-system openebs-lvm-node-gxwgr 2/2 Running 0 74s kube-system openebs-lvm-node-hkz69 2/2 Running 0 74s kube-system openebs-lvm-node-kv48x 2/2 Running 0 74s kube-system openebs-lvm-node-mqwm8 2/2 Running 0 74s kube-system openebs-lvm-node-tsxsl 2/2 Running 0 74s kube-system openebs-lvm-node-z2z6g 2/2 Running 0 74s |
|---|
创建 vg
集群中部分机器有 sdc ，手动在 sdc 上面创建 lvmvg 用于 openebs lvm local pv
1 2 | pvcreate /dev/sdc vgcreate lvmvg /dev/sdc |
|---|
创建 SC
因为 vg 只在部分机器上面有，所以需要使用 allowedTopologies 选项，指定可以使用的节点
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | cat > lvm-local-pv-sc.yaml <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv allowVolumeExpansion: true parameters: storage: "lvm" volgroup: "lvmvg" #上面创建的 vg 名 provisioner: local.csi.openebs.io allowedTopologies: - matchLabelExpressions: - key: kubernetes.io/hostname values: #有 lvmvg 的节点 - wx-vpck8s-worker098 - wx-vpck8s-worker099 - wx-vpck8s-worker100 - wx-vpck8s-worker101 - wx-vpck8s-worker102 EOF kubectl apply -f lvm-local-pv-sc.yaml |
|---|
部署应用
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | cat > lvm-local-pv-example.yaml <<EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv spec: storageClassName: openebs-lvmpv accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: Pod metadata: name: lvm-pod spec: restartPolicy: Never containers: - name: perfrunner image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done' volumeMounts: - mountPath: /datadir name: lvm-vol tty: true volumes: - name: lvm-vol persistentVolumeClaim: claimName: csi-lvmpv EOF kubectl apply -f lvm-local-pv-example.yaml |
|---|
查看应用和lvm信息
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # kubectl get pod,pv,pvc NAME READY STATUS RESTARTS AGE pod/lvm-pod 1/1 Running 0 27s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e 10Gi RWO Delete Bound default/csi-lvmpv openebs-lvmpv 26s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/csi-lvmpv Bound pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e 10Gi RWO openebs-lvmpv 27s # lvm 信息 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8 0 10G 0 lvm /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount |
|---|
性能测试
创建新的 pvc 用于性能测试
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | cat > dbench-pvc.yaml <<EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: dbench-lvm spec: storageClassName: openebs-lvmpv accessModes: - ReadWriteOnce resources: requests: storage: 50Gi EOF kubectl apply -f dbench-pvc.yaml |
|---|
下载性能测试文件：
performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark
调整以下内容：
1 2 3 4 | volumes: - name: dbench-lvm persistentVolumeClaim: claimName: dbench # 修改这里 |
|---|
创建应用
1 | kubectl create -f fio-deploy.yaml |
|---|
查看状态
1 2 3 4 | # kubectl get pod NAME READY STATUS RESTARTS AGE dbench-729c9--1-4clvm 1/1 Running 0 5s lvm-pod 1/1 Running 0 6m2s |
|---|
查看磁盘挂载信息
自动选择了另外一台机器的 sdc 创建分区挂载到 pod
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | 10.188.100.99 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--763113e0--c96f--4487--b3ae--9a306c159231 252:8 0 50G 0 lvm /var/lib/kubelet/pods/61686ab9-0538-4a89-a8e7-84ce9af351b1/volumes/kubernetes.io~csi/pvc-763113e0-c96f-4487-b3ae-9a306c159231/mount 10.188.100.98 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8 0 10G 0 lvm /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount |
|---|
查看性能测试结果
1 2 3 4 5 6 7 8 9 10 11 | kubectl logs dbench-729c9--1-4clvm …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 125k/23.9k. BW: 2623MiB/s / 962MiB/s Average Latency (usec) Read/Write: 95.08/385.63 Sequential Read/Write: 5042MiB/s / 1819MiB/s Mixed Random Read/Write IOPS: 23.7k/7938 |
|---|
清理
1 2 3 | kubectl delete -f fio-deploy.yaml kubectl delete -f dbench-pvc.yaml kubectl delete -f lvm-local-pv-example.yaml |
|---|
总结
三种模式pod磁盘性能差不多，lvm 模式下顺序读性能稍好。
- Local PV hostpath 需要手动挂载磁盘到对应目录
- Local PV device pod独占磁盘
- LVM Local PV 需要手动创建vg，pod 使用 lvm 较为灵活，目前推荐使用
参考文章：
https://weiliang-ms.github.io/wl-awesome/2.%E5%AE%B9%E5%99%A8/k8s/storage/OpenEBS.html