前提条件
说明
新建工作环境
$ mkdir <any-path>/dataset_scale$ cd <any-path>/dataset_scale
运行示例
创建 Dataset 和 GooseFSRuntime 资源对象
$ cat << EOF > dataset.yamlapiVersion: data.fluid.io/v1alpha1kind: Datasetmetadata:name: hbasespec:mounts:- mountPoint: https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/stable/name: hbase---apiVersion: data.fluid.io/v1alpha1kind: GooseFSRuntimemetadata:name: hbasespec:replicas: 1tieredstore:levels:- mediumtype: MEMpath: /dev/shmquota: 2Ghigh: "0.95"low: "0.7"EOF
说明
在上述示例中,我们设置
GooseFSRuntime.spec.replicas
为1,这意味着我们将启动一个带有一个 Worker 节点的 GooseFS 集群来缓存数据集中的数据。$ kubectl create -f dataset.yamldataset.data.fluid.io/hbase createdgoosefsruntime.data.fluid.io/hbase created
等待 GooseFS 集群正常启动后,可以看到此时创建完成的 Dataset 以及 GooseFSRuntime 处于如下状态:
GooseFS 各组件运行状态:
$ kubectl get podNAME READY STATUS RESTARTS AGEhbase-fuse-6pcnc 1/1 Running 0 3m15shbase-master-0 2/2 Running 0 3m50shbase-worker-w9wxh 2/2 Running 0 3m15s
Dataset 状态:
$ kubectl get dataset hbaseNAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGEhbase 544.77MiB 0.00B 2.00GiB 0.0% Bound 3m28s
GooseFSRuntime 状态:
$ kubectl get goosefsruntime hbase -o wideNAME READY MASTERS DESIRED MASTERS MASTER PHASE READY WORKERS DESIRED WORKERS WORKER PHASE READY FUSES DESIRED FUSES FUSE PHASE AGEhbase 1 1 Ready 1 1 Ready 1 1 Ready 4m55s
Dataset 扩容
$ kubectl scale goosefsruntime hbase --replicas=2goosefsruntime.data.fluid.io/hbase scaled
直接使用
kubectl scale
命令即可完成 Dataset 的扩容操作。在成功执行上述命令并等待一段时间后可以看到 Dataset 以及 GooseFSRuntime 的状态均发生了变化:一个新的 GooseFS Worker 以及对应的 GooseFS Fuse 组件成功启动:
$ kubectl get podNAME READY STATUS RESTARTS AGEhbase-fuse-6pcnc 1/1 Running 0 13mhbase-fuse-8qgww 1/1 Running 0 6m49shbase-master-0 2/2 Running 0 13mhbase-worker-l4c8n 2/2 Running 0 6m49shbase-worker-w9wxh 2/2 Running 0 13m
Dataset 中的
Cache Capacity
从原来的 2.00GiB
变为 4.00GiB
,表明该 Dataset 的可用缓存容量增加:$ kubectl get dataset hbaseNAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGEhbase 544.77MiB 0.00B 4.00GiB 0.0% Bound 15m
GooseFSRuntime 中的
Ready Workers
以及 Ready Fuses
属性均变为2:$ kubectl get goosefsruntime hbase -o wideNAME READY MASTERS DESIRED MASTERS MASTER PHASE READY WORKERS DESIRED WORKERS WORKER PHASE READY FUSES DESIRED FUSES FUSE PHASE AGEhbase 1 1 Ready 2 2 Ready 2 2 Ready 17m
查看 GooseFSRuntime 的具体描述信息可以了解最新的扩缩容信息:
$ kubectl describe goosefsruntime hbase...Conditions:...Last Probe Time: 2021-04-23T07:54:03ZLast Transition Time: 2021-04-23T07:54:03ZMessage: The workers are scale out.Reason: Workers scaled outStatus: TrueType: Workers scaled outLast Probe Time: 2021-04-23T07:54:03ZLast Transition Time: 2021-04-23T07:54:03ZMessage: The fuses are scale out.Reason: Fuses scaled outStatus: TrueType: FusesScaledOut...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Succeed 2m2s GooseFSRuntime GooseFS runtime scaled out. current replicas: 2, desired replicas: 2.
Dataset 缩容
与扩容类似,缩容时同样可以使用
kubectl scale
对 Runtime 的 Worker 数量进行调整:$ kubectl scale goosefsruntime hbase --replicas=1goosefsruntime.data.fluid.io/hbase scaled
成功执行上述命令后,如果目前环境中没有应用正在尝试访问该数据集,那么就会触发 Runtime 的缩容。
超出指定
replicas
数量的 Runtime Worke r将会被停止:NAME READY STATUS RESTARTS AGEhbase-fuse-8qgww 1/1 Running 0 21mhbase-fuse-zql96 1/1 Terminating 0 17m32shbase-master-0 2/2 Running 0 22mhbase-worker-f92vv 2/2 Terminating 0 17m32shbase-worker-l4c8n 2/2 Running 0 21m
Dataset 的缓存容量
Cache Capacity
恢复到 2.00GiB
:$ kubectl get dataset hbaseNAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGEhbase 544.77MiB 0.00B 2.00GiB 0.0% Bound 30m
注意
在目前版本的 Fluid 中,缩容时 Dataset 中
Cache Capacity
属性字段的变化存在几分钟的延迟,因此您可能无法迅速观察到这一属性的变化。GooseFSRuntime 中的
Ready Workers
以及 Ready Fuses
字段同样变为 1
:$ kubectl get goosefsruntime hbase -o wideNAME READY MASTERS DESIRED MASTERS MASTER PHASE READY WORKERS DESIRED WORKERS WORKER PHASE READY FUSES DESIRED FUSES FUSE PHASE AGEhbase 1 1 Ready 1 1 Ready 1 1 Ready 30m
查看 GooseFSRuntime 的具体描述信息可以了解最新的扩缩容信息:
$ kubectl describe goosefsruntime hbase...Conditions:...Last Probe Time: 2021-04-23T08:00:55ZLast Transition Time: 2021-04-23T08:00:55ZMessage: The workers scaled in.Reason: Workers scaled inStatus: TrueType: WorkersScaledInLast Probe Time: 2021-04-23T08:00:55ZLast Transition Time: 2021-04-23T08:00:55ZMessage: The fuses scaled in.Reason: Fuses scaled inStatus: TrueType: FusesScaledIn...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Succeed 6m56s GooseFSRuntime GooseFS runtime scaled out. current replicas: 2, desired replicas: 2.Normal Succeed 4s GooseFSRuntime GooseFS runtime scaled in. current replicas: 1, desired replicas: 1.
Fluid 提供的这种扩缩容能力,能够帮助用户或是集群管理员适时地调整数据集缓存所占用的集群资源,减少某个不频繁使用的数据集的缓存容量(缩容),或者按需增加某数据集的缓存容量(扩容),以实现更加精细的资源分配,提高资源利用率。
环境清理
$ kubectl delete -f dataset.yaml