外部存储接入 Kubernetes 的方式主要有两种:In-Tree 和 Out-of-Tree:
外部存储最终的效果是将存储(磁盘、obs、nas盘等)挂载到容器中被业务使用,所以一般包括存在两个过程:
kubernetes CSI存储插件的关键组件与推荐的容器化部署架构
以块类型存储为例,从声明pvc到pod挂载卷成功时序图:
一、涉及组件解读
二、涉及资源解读
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-test
labels:
failure-domain.beta.kubernetes.io/region: cn-north-4
failure-domain.beta.kubernetes.io/zone: cn-north-4a
annotations:
pv.kubernetes.io/provisioned-by: xxxx-provisioner #存储提供者
spec:
capacity:
storage: 10Gi
csi:
driver: disk.csi.everest.io
volumeHandle: 698a99d8-xxx-xxxx-xxxx-ab80b1ecbf #使用的存储设备信息
volumeAttributes:
everest.io/disk-mode: SCSI
everest.io/disk-volume-type: ESSD
storage.kubernetes.io/csiProvisionerIdentity: xxxx-provisioner
accessModes:
- ReadWriteOnce
# 引用对象, 该pv由哪个pvc创建
claimRef:
kind: PersistentVolumeClaim
namespace: test
name: pvc-test
uid: xxxx-xxxx-xxxx-22bf9101f0ce
apiVersion: v1
persistentVolumeReclaimPolicy: Delete
storageClassName: csi-disk
volumeMode: Filesystem
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- cn-north-4a
status:
phase: Bound
# available : 表示当前的pv没有被绑定
# bound: 已经被pvc挂载
# released: pvc没有在使用pv, 需要管理员手工释放pv
# failed: 资源回收失败
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-evs-test
namespace: test
uid: xxxx-xxxx-xxxx-22bf9101f0ce
labels:
failure-domain.beta.kubernetes.io/region: cn-north-4
failure-domain.beta.kubernetes.io/zone: cn-north-4a
annotations:
volume.kubernetes.io/selected-node: xxx.xxx.xxx.186
everest.io/disk-volume-type: ESSD
volume.kubernetes.io/storage-provisioner: xxxx-provisioner
spec:
# ReadWriteOnce:被单个节点mount为读写rw模式
# ReadOnlyMany 被多个节点mount为只读ro模式
# ReadWriteMany 被多个节点mount为读写rw模式
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeName: pv-test #绑定的pv name
# 使用的sc类型
storageClassName: csi-disk
# 存储模式,包含Filesystem(文件系统)和Block(块设备)
volumeMode: Filesystem
status:
# Pending:pvc刚创建还未与pv绑定
# Bound:pvc与pv完成绑定
# Lost:对应的pv被删除
phase: Bound
accessModes:
- ReadWriteOnce
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-disk
parameters:
csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
csi.storage.k8s.io/fstype: ext4
everest.io/disk-volume-type: SATA
everest.io/passthrough: "true"
provisioner: xxxx-provisioner
# 回收策略, pvc和pv解绑,删除了pvc, pv里面的数据是否还保留
# Retain: 保留数据, 需要手工删除
# delete: pv删除
reclaimPolicy: Delete
# Immediate: pv创建好之后立马将pvc和pv进行绑定
# WaitForFirstConsumer: 延迟绑定,直到使用pvc的pod被调度到节点上
volumeBindingMode: Immediate
allowVolumeExpansion: true #是否允许扩容
三、涉及 CSI API 对象
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
everest.io/node.localvolume.capacity: "null"
name: xxx.xxx.xxx.186
ownerReferences:
- apiVersion: v1
kind: Node
name: xxx.xxx.xxx.186
uid: 091cc415-b8bb-4173-8312-5f6318d4383f
uid: fea2c180-99b8-4195-a966-3953b8bab16a
spec:
# 节点上有哪些driver
drivers:
- allocatable:
count: 58
name: disk.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys:
- failure-domain.beta.kubernetes.io/zone
- name: proxy.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
- name: sfsturbo.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
- name: nas.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
...
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
everest.io/node.localvolume.capacity: "null"
name: xxx.xxx.xxx.186
ownerReferences:
- apiVersion: v1
kind: Node
name: xxx.xxx.xxx.186
uid: 091cc415-b8bb-4173-8312-5f6318d4383f
uid: fea2c180-99b8-4195-a966-3953b8bab16a
spec:
# 节点上有哪些driver
drivers:
- allocatable:
count: 58
name: disk.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys:
- failure-domain.beta.kubernetes.io/zone
- name: proxy.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
- name: sfsturbo.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
- name: nas.csi.everest.io
nodeID: 7d279bf8-c70f-4179-842e-5e501d591d17
topologyKeys: null
...
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: disk.csi.everest.io
uid: 5d33a29b-4bf1-4ab8-815f-e97b207b991e
spec:
# 是否需要attache和mount,只有evs需要attach
attachRequired: true
podInfoOnMount: true
requiresRepublish: false
storageCapacity: false
volumeLifecycleModes:
- Persistent #volume生命周期,持久模式
AD Controller 创建一个 VolumeAttachment,而 External-attacher 则通过观察该 VolumeAttachment,根据其状态属性来进行存储的挂载和卸载操作。
apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
annotations:
csi.alpha.kubernetes.io/node-id: xxxx-xxxx-xxxx-5e501d591d17
finalizers:
- everest-csi-attacher/disk-csi-everest-io
name: csi-d10b9f7e4dde469fa2b7f3461fcfef7862260883196647d6b7ae7bb17bc0e226
uid: 665b740f-a544-4f3e-9953-00b8d186c548
spec:
attacher: disk.csi.everest.io
nodeName:xxx.xxx.xxx.186
source:
persistentVolumeName: pv-test
status:
# 标记是否attached到节点上,attache后才能mount
attached: true
attachmentMetadata: #attach的设备信息
bus: scsi
device: /dev/sdg
kubernetes里面有两个绑定:
stroageclass延迟绑定作用字段:VolumeBindingMode