注:使用源码安装的原因主要是使用yum安装glusterfs服务端时出现一些依赖库问题
10.85.3.113 glusterfs-1.example.com
10.85.3.114 glusterfs-2.example.com
10.85.3.115 glusterfs-3.example.com
# yum install autoconf automake bison cmockery2-devel dos2unix flex fuse-devel glib2-devel libacl-devel libaio-devel libattr-devel libcurl-devel libibverbs-devel librdmacm-devel libtirpc-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig pyliblzma python-devel python-eventlet python-netifaces python-paste-deploy python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build sqlite-devel systemtap-sdt-devel tar userspace-rcu-devel
# cd userspace-rcu-master
# ./bootstrap
# ./configure
# make && make install
# ldconfig
# cd /home/glusterfs-5.7
# cp -r libglusterfs/src /usr/local/include/glusterfs
# cp -r /home/glusterfs-xlators-master/xlators/features/glupy/ /home/glusterfs-5.7/xlators/features/
# cd /home/glusterfs-5.7
# ./autogen.sh
# ./configure --without-libtirpc
# ./configure --enable-gnfs
# make
# make install
在make的时候可能会遇到如下错误
../../../../contrib/userspace-rcu/rculist-extra.h:33:6: error: redefinition of 'cds_list_add_tail_rcu'
void cds_list_add_tail_rcu(struct cds_list_head *newp,
^
In file included from glusterd-rcu.h:15:0,
from glusterd-sm.h:26,
from glusterd.h:28,
from glusterd.c:19:
/usr/local/include/urcu/rculist.h:44:6: note: previous definition of 'cds_list_add_tail_rcu' was here
void cds_list_add_tail_rcu(struct cds_list_head *newp,
给下述文件的cds_list_add_tail_rcu函数加上条件编译即可
/usr/local/include/urcu/rculist.h
/home/glusterfs-5.7/contrib/userspace-rcu/rculist-extra.h
#ifndef CDS_LIST_ADD_TAIL_CRU
#define CDS_LIST_ADD_TAIL_CRU
static inline
void cds_list_add_tail_rcu(struct cds_list_head *newp,
struct cds_list_head *head)
{
newp->next = head;
newp->prev = head->prev;
rcu_assign_pointer(head->prev->next, newp);
head->prev = newp;
}
#endif
mkdir -p /data/brick
chown -R root:65534 /data
chmod -R 775 /data
vgcreate vgglusterfs /dev/vdb
lvcreate -L 499G -n glusterfs vgglusterfs
mkfs.xfs /dev/mapper/vgglusterfs-glusterfs
#/etc/fstab中添加:/dev/mapper/vgglusterfs-glusterfs /data/brick xfs defaults 0 0
mount -a
systemctl enable glusterd.service
使用如下命令查看生成的gluster.service的路径,本系统为:/usr/local/lib/systemd/system/glusterd.service
systemctl cat glusterd.service
/usr/local/lib/systemd/system/glusterd.service的内容如下
[Unit]
Description=GlusterFS, a clustered file-system server
Requires=rpcbind.service
After=network.target rpcbind.service
Before=network-online.target
[Service]
Type=forking
PIDFile=/var/run/glusterd.pid
LimitNOFILE=1048576
Environment="LOG_LEVEL=INFO"
ExecStart=/usr/local/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS
KillMode=process
SuccessExitStatus=15
[Install]
WantedBy=multi-user.target
执行如下命令启动
systemctl daemon-reload
systemctl start glusterd.service
如果遇到“Failed to restart glusterd.service: Unit not found”错误,可能是rpcbind.service没有安装,执行yum install -y rpcbind安装即可
遇到“Failed to start GlusterFS, a clustered file-system server”错误,可以尝试如下方式:
# gluster peer probe glusterfs-1.example.com
# gluster peer probe glusterfs-2.example.com
# gluster peer probe glusterfs-3.example.com
# gluster peer status
gluster volume create volume-5G replica 3 glusterfs-1.example.com:/data/brick/glusterfs1 glusterfs-2.example.com:/data/brick/glusterfs2 glusterfs-3.example.com:/data/brick/glusterfs3
gluster volume start volume-5G
使用如下命令查看volume
gluster volume status
gluster volume info
容器环境下推荐Replicated 模式,更多信息参见官方文档,注意部分模式已经废弃
# yum install glusterfs-client -y
mount -t glusterfs glusterfs-1.example.com:/volume-5G /data/mounttest/
glusterfs --volfile-id=volume-5G --volfile-server=glusterfs-1.example.com /data/glusterfsclient
# netstat -ntp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.85.3.111:1015 10.85.3.115:49152 ESTABLISHED 10679/glusterfs
tcp 0 0 10.85.3.111:1017 10.85.3.113:49152 ESTABLISHED 10679/glusterfs
tcp 0 0 10.85.3.111:1023 10.85.3.113:24007 ESTABLISHED 10679/glusterfs
tcp 0 0 10.85.3.111:1016 10.85.3.114:49152 ESTABLISHED 10679/glusterfs
TIPS:
# rm -rf /data/brick/glusterfs1/*
# setfattr -x trusted.glusterfs.volume-id /data/brick/glusterfs1/
# 开启 指定 volume 的配额
$ gluster volume quota k8s-volume enable
# 限制 指定 volume 的配额
$ gluster volume quota k8s-volume limit-usage / 1TB
# 设置 cache 大小, 默认32MB
$ gluster volume set k8s-volume performance.cache-size 4GB
# 设置 io 线程, 太大会导致进程崩溃
$ gluster volume set k8s-volume performance.io-thread-count 16
# 设置 网络检测时间, 默认42s
$ gluster volume set k8s-volume network.ping-timeout 10
# 设置 写缓冲区的大小, 默认1M
$ gluster volume set k8s-volume performance.write-behind-window-size 1024MB
volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... <start|stop|status|commit|force>
volume add-brick <VOLNAME> [<stripe|replica> <COUNT> [arbiter <COUNT>]] <NEW-BRICK> ... [force]
下载fio,解压后执行“make & make install”即可。可能需要执行"yum install libaio-devel"
在一台服务器上下载并执行“make build”编译gluster_exporter,直接使用./gluster_exporter启动即可。(为保证node节点有效性,最好在每台服务器上安装node_exporter)。使用如下方式可以简单启动一个exporter,监听端口默认是9189
nohup gluster_exporter --web.telemetry-path="/usr/local/sbin/gluster" 2>&1 &
在Prometheus中添加新job,即可使用Prometheus监控glusterfs
- job_name: glusterfs1
static_configs:
- targets: ['10.85.3.113:9189']
labels:
alias: glusterfs1
VOLNAME
clients可以查看volume的client信息目前kubernetes使用glusterfs storageclass时需要用到heketi,下面讲解heketi的部署
在官网下载heketi压缩包heketi-vx.x.x.linux.amd64.tar.gz,可以按照官网文档进行安装部署,虚机部署方式如下:
mkdir /etc/heketi/
mkdir /var/lib/heketi/
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "22",
"fstab": "Optional: Specify fstab file on node. Default is /etc/fstab",
"pv_data_alignment": "Optional: Specify PV data alignment size. Default is 256K",
"vg_physicalextentsize": "Optional: Specify VG physical extent size. Default is 4MB",
"lv_chunksize": "Optional: Specify LV chunksize. Default is 256K",
"backup_lvm_metadata": false,
"_debug_umount_failures": "Optional: boolean to capture more details in case brick unmounting fails",
"debug_umount_failures": true
},
ssh-keygen -t rsa -q -f /etc/heketi/heketi_key -N ''
chmod 775 /etc/heketi/heketi_key
在所有gluasterfs服务器上为heketi添加ssh互信
cat heketi_key.pub >> /root/.ssh/authorized_keys
[Unit]
Description=Heketi Server
[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.json
User=heketi
ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
并在/etc/systemd/system/multi-user.target.wants中创建软连接
ln -s /usr/lib/systemd/system/heketi.service heketi.service
加载并启动heketi
systemctl daemon-reload
systemctl enable heketi
systemctl start heketi
如果启动时出现如下错误,可在/etc/heketi/heketi.json中删除对应的行即可,如出现如下错误,删除"xfs_sw"对应的行,使用默认值即可。(建议配置或删除所有“Optional”的行)
ERROR: Unable to parse configuration: json: cannot unmarshal string into Go struct field KubeConfig.xfs_sw of type int
$ export HEKETI_CLI_SERVER=http://<heketi server and port>
$ heketi-cli topology load --json=<topology>
拓扑文件内容如下,需要注意的是,heketi只能使用裸磁盘,不能使用文件系统以及lvm
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"10.85.3.113"
],
"storage": [
"10.85.3.113"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/vdb",
"destroydata": false
}
]
},
{
"node": {
"hostnames": {
"manage": [
"10.85.3.114"
],
"storage": [
"10.85.3.114"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/vdb",
"destroydata": false
}
]
},
{
"node": {
"hostnames": {
"manage": [
"10.85.3.115"
],
"storage": [
"10.85.3.115"
]
},
"zone": 2
},
"devices": [
{
"name": "/dev/vdb",
"destroydata": false
}
]
}
]
}
]
}
heketi-cli volume create --size=10 --name="test"
可以看到新创建的volume
# heketi-cli volume list
Id:0a0e0fd1f81973044bb0365e44c08648 Cluster:b988c638edcef9b5b44e0e42ccef30b7 Name:tes
在glusterfs上查看可以看到heketi其实也是使用了devicemapper
# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/vg_c17870f6f2870ff1c8001ad19b3b9d5b-brick_e3e02d5357b540ce340ec5328fe8b0ec 10G 33M 10G 1% /var/lib/heketi/mounts/vg_c17870f6f2870ff1c8001ad19b3b9d5b/brick_e3e02d5357b540ce340ec5328fe8b0ec
当然此时也可以使用gluster命令查看volume
# gluster volume list
test
heketi命令行一般如下
export HEKETI_CLI_SERVER=http://<heketi server and port>
heketi-cli cluster list
heketi-cli cluster info $ID
heketi-cli node list
heketi-cli node info $ID
heketi-cli topology info
heketi-cli volume create --size=10 //10G
heketi-cli volume delete $ID
heketi-cli device disable $ID
keheti-cli device remove $ID
heketi-cli device delete $ID
注意:使用heketi管理volume后,仅使用heketi,不能glusterfs和heketi混用
首先创建storageclass
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: glusterfs
namespace: demo
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.86.3.113:18080"
restuser: "root"
restauthenabled: "false"
然后创建pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: glusterfs-test
namespace: demo
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30Gi
storageClassName: glusterfs
需要注意的是如果heketi topology文件中使用了域名,则该域名必须能够被kubernetes解析,否则会失败
TIPS:
Failed to provision volume with StorageClass "glusterfs": glusterfs: create volume err: failed to create endpoint/service <nil>
systemctl stop heketi
//在所有glusterfs服务器上执行如下操作
umount /var/lib/heketi/mounts/vg_xxxxxxx/brick_xxxx
rm -rf /var/lib/heketi/*
删除heketi创建的lv和vg
参考:
https://www.cnblogs.com/jicki/p/5801712.html
https://www.ibm.com/developerworks/cn/opensource/os-cn-glusterfs-docker-volume/index.html
https://jimmysong.io/kubernetes-handbook/practice/using-glusterfs-for-persistent-storage.html
https://pdf.us/2019/03/15/3020.html
https://github.com/psyhomb/heketi
https://github.com/heketi/heketi/blob/master/docs/admin/volume.md