kubernetes中用NFS做后端存储支不支持PVC扩容？

极客运维圈

发布于 2020-03-23 16:10:42

4.7K1

发布于 2020-03-23 16:10:42

文章被收录于专栏：乔边故事乔边故事

在虚拟机时代，我们经常会收到给磁盘扩容的需求，一般我们就是添加一块磁盘，然后做LVM扩容操作，那在容器时代，我们可以给PVC扩容吗？

之前发了持久化存储相关的知识--PV和PVC，StorageClass，本文来给你介绍一下用NFS做后端存储能不能扩容的问题，当然在生产上没有哪个愿意用NFS做后端存储......

你首先需要准备好NFS存储

首先我们介绍一下pv和pvc模式，也就是纯手动创建的模式。（1）、首先我们创建一个PV的YAML文件（pv2.yaml）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv02
  labels:
    storage: pv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 500Mi
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/k8s
    server: 10.1.10.128

创建pv

# kubectl apply -f pv2.yaml
persistentvolume/my-pv02 created
#  kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM              STORAGECLASS   REASON   AGE                                        30s
my-pv02   500Mi      RWO            Recycle          Bound       default/pvc-test                           30s

（2）、创建PVC（pvc.yaml），我们这里创建一个比较小的pvc，如下

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

然后创建pvc

# kubectl apply -f pvc.yaml
# kubectl get pvc
NAME       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-test   Bound    my-pv02   500Mi      RWO                           58s

我们其实可以看到，我在pvc里定义的是500Mi，它会自动匹配刚好合适的PV。这时候如果我们修改my-pv02的YAML文件，如下：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv02
  labels:
    storage: pv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/k8s
    server: 10.1.10.128

然后我们尝试创建这个PV，如下：

# kubectl apply -f pv2.yaml
persistentvolume/my-pv02 configured
# kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM              STORAGECLASS   REASON   AGE
my-pv02   2Gi        RWO            Recycle          Bound       default/pvc-test                           116m
# kubectl get pvc
NAME       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-test   Bound    my-pv02   500Mi      RWO                           116m

我们看到我们my-pv02的CAPACITY变成了2Gi，我们的pvc还是500Mi，那么我们能不能对PVC进行扩容呢？我尝试的修改了pvc的配置文件，如下：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

然后我们执行这个配置文件

]# kubectl apply -f  pvc.yaml
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"pvc-test\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"1Gi\"}}}}\n"}},"spec":{"resources":{"requests":{"storage":"1Gi"}}}}
to:
Resource: "/v1, Resource=persistentvolumeclaims", GroupVersionKind: "/v1, Kind=PersistentVolumeClaim"
Name: "pvc-test", Namespace: "default"
Object: &{map["apiVersion":"v1" "kind":"PersistentVolumeClaim" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"pvc-test\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"500Mi\"}}}}\n" "pv.kubernetes.io/bind-completed":"yes" "pv.kubernetes.io/bound-by-controller":"yes"] "creationTimestamp":"2019-12-24T10:46:05Z" "finalizers":["kubernetes.io/pvc-protection"] "name":"pvc-test" "namespace":"default" "resourceVersion":"15059" "selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/pvc-test" "uid":"6a905af6-5e44-4940-ba87-e56f6e99df24"] "spec":map["accessModes":["ReadWriteOnce"] "resources":map["requests":map["storage":"500Mi"]] "volumeMode":"Filesystem" "volumeName":"my-pv02"] "status":map["accessModes":["ReadWriteOnce"] "capacity":map["storage":"500Mi"] "phase":"Bound"]]}
for: "pvc.yaml": persistentvolumeclaims "pvc-test" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

我们发现报错了，错误信息如下：

persistentvolumeclaims "pvc-test" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

它的意思就是说只有动态供应的pvc可以调整大小，供应pvc的存储类必须支持调整大小。也就是说我们手动创建PV，PVC模式是不支持调整大小的。

我们现在来验证一下用storageclass创建的PVC。（1）、因为我们使用nfs作为后端存储，所以我们需要先创建一个nfs-client，YAML文件如下：

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nfs-client-provisioner-clusterrole
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nfs-client-provisioner-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: nfs-client-provisioner
  namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-clusterrole
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-client-prosioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-prosioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
      - name: nfs-client-prosioner
        image: registry.cn-hangzhou.aliyuncs.com/rookieops/nfs-client-provisioner:v0.1
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: nfs-client-root
          mountPath: /data/pv
        env:
        - name: PROVISIONER_NAME
          value: rookieops/nfs
        - name: NFS_SERVER
          value: 10.1.10.128
        - name: NFS_PATH
          value: /data/k8s
      volumes:
      - name: nfs-client-root
        nfs:
          server: 10.1.10.128
          path: /data/k8s

创建YAML文件

# kubectl apply -f !$
kubectl apply -f nfs-client.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/nfs-client-provisioner-clusterrolebinding created
deployment.extensions/nfs-client-prosioner created
# kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
nfs-client-prosioner-56f44c675b-t2ml9   1/1     Running   0          6s

（1）我们创建一个普通的StorageClass（storageclass.yaml）

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client-storageclass
provisioner: rookieops/nfs

创建这个YAML文件

# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/nfs-client-storageclass created
# kubectl get storageclasses.storage.k8s.io
NAME                      PROVISIONER     AGE
nfs-client-storageclass   rookieops/nfs   34s

创建一个PVC（storageclass-pvc.yaml）

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nfs-pvc2
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs-client-storageclass"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

创建这个YAML文件

# kubectl apply -f storageclass-pvc.yaml
persistentvolumeclaim/test-nfs-pvc2 created
# kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
pvc-test        Bound    my-pv02                                    500Mi      RWO                                      26d
test-nfs-pvc2   Bound    pvc-9dd9c608-6875-4609-829c-c92920aaa783   1Mi        RWX            nfs-client-storageclass   21s

我们看到已经Bound了，我们现在直接修改pvc（storageclass-pvc.yaml）

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nfs-pvc2
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs-client-storageclass"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi

然后我们执行这个YAML文件

# kubectl apply -f storageclass-pvc.yaml
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"volume.beta.kubernetes.io/storage-class\":\"nfs-client-storageclass\"},\"name\":\"test-nfs-pvc2\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteMany\"],\"resources\":{\"requests\":{\"storage\":\"100Mi\"}}}}\n"}},"spec":{"resources":{"requests":{"storage":"100Mi"}}}}
to:
Resource: "/v1, Resource=persistentvolumeclaims", GroupVersionKind: "/v1, Kind=PersistentVolumeClaim"
Name: "test-nfs-pvc2", Namespace: "default"
Object: &{map["apiVersion":"v1" "kind":"PersistentVolumeClaim" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"volume.beta.kubernetes.io/storage-class\":\"nfs-client-storageclass\"},\"name\":\"test-nfs-pvc2\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteMany\"],\"resources\":{\"requests\":{\"storage\":\"1Mi\"}}}}\n" "pv.kubernetes.io/bind-completed":"yes" "pv.kubernetes.io/bound-by-controller":"yes" "volume.beta.kubernetes.io/storage-class":"nfs-client-storageclass" "volume.beta.kubernetes.io/storage-provisioner":"rookieops/nfs"] "creationTimestamp":"2020-01-20T06:04:20Z" "finalizers":["kubernetes.io/pvc-protection"] "name":"test-nfs-pvc2" "namespace":"default" "resourceVersion":"29483" "selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/test-nfs-pvc2" "uid":"9dd9c608-6875-4609-829c-c92920aaa783"] "spec":map["accessModes":["ReadWriteMany"] "resources":map["requests":map["storage":"1Mi"]] "volumeMode":"Filesystem" "volumeName":"pvc-9dd9c608-6875-4609-829c-c92920aaa783"] "status":map["accessModes":["ReadWriteMany"] "capacity":map["storage":"1Mi"] "phase":"Bound"]]}
for: "storageclass-pvc.yaml": persistentvolumeclaims "test-nfs-pvc2" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

发现报与上面一样的错，这是因为要支持动态扩容需要满足两个条件：

后端底层存储支持卷扩展（后端存储保证足够资源）
需要在StorageClass对象中设置allowVolumeExpansion为true

我们由于是测试，申请的资源比较少，我们直接对StorageClass对象进行修改（storageclass.yaml），如下：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client-storageclass
provisioner: rookieops/nfs
allowVolumeExpansion: true

然后我们重新申明一下

# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/nfs-client-storageclass configured

我们再次创建前面的storageclass-pvc.yaml文件

# kubectl apply -f storageclass-pvc.yaml
persistentvolumeclaim/test-nfs-pvc2 configured

我们发现这次没有报错，我们查看一下这个PVC

# kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
pvc-test        Bound    my-pv02                                    500Mi      RWO                                      26d
test-nfs-pvc2   Bound    pvc-9dd9c608-6875-4609-829c-c92920aaa783   1Mi        RWX            nfs-client-storageclass   14m

我们一看，发现并没有扩展成功，这是为什么呢？我们describe一下这个pvc，发现如下提示：

# kubectl describe pvc test-nfs-pvc2
...
Warning  ExternalExpanding      7m43s  volume_expand
Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.

报错信息为：没有找到可扩展的插件。我上官方网站一看，原来人家已经说的很清楚了：Although the feature is enabled by default, a cluster admin must opt-in to allow users to resize their volumes. Kubernetes v1.11 ships with volume expansion support for the following in-tree volume plugins: AWS-EBS, GCE-PD, Azure Disk, Azure File, Glusterfs, Cinder, Portworx, and Ceph RBD.

我们的NFS并不被支持（用NFS做后端存储的小伙伴注意了哈）。

本文参与腾讯云自媒体同步曝光计划，分享自微信公众号。

原始发表：2020-02-07，如有侵权请联系 cloudcommunity@tencent.com 删除

存储