ceph块设备使用iscsi

summerking

发布于 2023-04-27 20:36:51

1.2K00

代码可运行

运行总次数：0

代码可运行

Ceph 块设备（也叫RBD）服务提供了大小可调、精炼、支持快照和克隆的块设备。为提供高性能， Ceph 把块设备条带化到整个集群。 Ceph 同时支持内核对象（ KO ）和 QEMU 管理程序直接使用 librbd ——避免了内核对象在虚拟系统上的开销。

在linux上使用iscsi挂载块设备

1. 确认iscsi-initiator-utils已安装

yum -y install scsi-target-utils

[root@summer163-200 ~]# rpm -qa |grep -i iscsi-initiator-utils
iscsi-initiator-utils-iscsiuio-6.2.0.874-19.0.1.el7.x86_64
iscsi-initiator-utils-6.2.0.874-19.0.1.el7.x86_64

2. 配置文件

root in summer163-200 in ~ via 🐍 v2.7.5 
➜ sed -n '50,90p'  /etc/iscsi/iscsid.conf | grep -Ev '^$|^\s*#'
node.session.auth.authmethod = CHAP
node.session.auth.username = xsw1
node.session.auth.password = q123456123456
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = xsw1
discovery.sendtargets.auth.password = q123456123456

[root@summer163-200 ~]# sed -n '50,90p'  /etc/iscsi/iscsid.conf

# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP

# To configure which CHAP algorithms to enable set
# node.session.auth.chap_algs to a comma seperated list.
# The algorithms should be listen with most prefered first.
# Valid values are MD5, SHA1, SHA256
# The default is MD5.
#node.session.auth.chap_algs = SHA256,SHA1,MD5

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = xsw1
node.session.auth.password = q123456123456

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = xsw1
#node.session.auth.password_in = q123456123456

# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
discovery.sendtargets.auth.username = xsw1
discovery.sendtargets.auth.password = q123456123456

# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = xsw1
#discovery.sendtargets.auth.password_in = q123456123456

# ********

3. 重启服务

[root@summer163-200 ~]# service iscsid restart
Redirecting to /bin/systemctl restart iscsid.service
[root@summer163-200 ~]# ps -ef |grep iscsi
root      353907       2  0 Mar24 ?        00:00:00 [iscsi_eh]
root      811588       1  0 09:46 ?        00:00:00 /sbin/iscsid -f -d2
root      811603  808540  0 09:46 pts/2    00:00:00 grep --color=auto iscsi

4. 发现目标

这里注意添加ACL,不配置默认是ALL
什么是ACL？ ACL（Access Control List）即访问控制列表。主要是针对单一用户，单一文件或目录进行rwx权限的细部设定。可以针对用户（User）、群组（Group）、默认属性掩码（umask）进行设置。

root in summer163-200 in /var/lib/iscsi 
➜ iscsiadm -m discovery -t st -p 192.168.163.201
192.168.163.201:3260,1 iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c

在发现了target后，在/var/lib/iscsi中的nodes和send_targets目录中就有了文件

root in summer163-200 in /var/lib 
➜ iscsiadm -m discovery -t st -p 192.168.163.299
iscsiadm: Cannot resolve host 192.168.163.299. getaddrinfo error: [Name or service not known]
iscsiadm: cannot resolve host name 192.168.163.299
iscsiadm: cannot resolve host name 192.168.163.299
iscsiadm: Could not perform SendTargets discovery: encountered connection failure
root in summer163-200 in /var/lib 
❯ tree iscsi/
iscsi/
├── ifaces
├── isns
├── nodes
│   └── iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c
│       └── 192.168.163.201,3260,1
│           └── default
├── send_targets
│   ├── 192.168.163.201,3260
│   │   ├── iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c,192.168.163.201,3260,1,default -> /var/lib/iscsi/nodes/iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c/192.168.163.201,3260,1
│   │   └── st_config
│   └── 192.168.163.299,3260
│       └── st_config
├── slp
└── static

11 directories, 3 files

discovery失败的在send_targets目录中也会记录，只不过是空目录。在nodes目录和正确discovery到的send_targets子目录中都有几个配置文件，都是些参数信息。而且可以看到，send_targets中的目录是nodes目录的软链接
如果想重新发现已存在的target时，可以清空nodes目录中对应的项，然后再discovery

5. 登入目标

root in summer163-200 in iscsi/send_targets/192.168.163.299,3260 
➜ iscsiadm -m node iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c -l
Logging in to [iface: default, target: iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c, portal: 192.168.163.201,3260] (multiple)
Login to [iface: default, target: iqn.2021-04.com.419:192.168.163.sn.2f0ab0113d94581903dd010c, portal: 192.168.163.201,3260] successful.

6. 查看iscsi映射的磁盘和其分区情况

[root@summer163-200 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
├─sda2        8:2    0    5G  0 part /usr/summer_vfs
└─sda3        8:3    0   94G  0 part 
  └─os-root 252:0    0   94G  0 lvm  /
sdb           8:16   0   30G  0 disk 
└─sdb1        8:17   0   30G  0 part /var/lib/ceph/osd/ceph-0
sdc           8:32   0   30G  0 disk 
└─sdc1        8:33   0   30G  0 part /var/lib/ceph/osd/ceph-1
sdd           8:48   0   10G  0 disk 
├─sdd1        8:49   0   16M  0 part 
└─sdd2        8:50   0   10G  0 part 
sr0          11:0    1 1024M  0 rom

7. 重新分区

[root@summer163-200 ~]# parted -s /dev/sdd mklabel gpt
[root@summer163-200 ~]# parted -s /dev/sdd mkpart primary 0G 100%
[root@summer163-200 ~]# parted -s /dev/sdd print
Model: IET VIRTUAL-DISK (scsi)
Disk /dev/sdd: 10.7GB
Sector size (logical/physical): 512B/4194304B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  10.7GB  10.7GB               primary

[root@summer163-200 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
├─sda2        8:2    0    5G  0 part /usr/summer_vfs
└─sda3        8:3    0   94G  0 part 
  └─os-root 252:0    0   94G  0 lvm  /
sdb           8:16   0   30G  0 disk 
└─sdb1        8:17   0   30G  0 part /var/lib/ceph/osd/ceph-0
sdc           8:32   0   30G  0 disk 
└─sdc1        8:33   0   30G  0 part /var/lib/ceph/osd/ceph-1
sdd           8:48   0   10G  0 disk 
└─sdd1        8:49   0   10G  0 part 
sr0          11:0    1 1024M  0 rom

8. 创建文件系统并挂载块设备

[root@summer163-200 ~]# mkfs.xfs /dev/sdd1
specified blocksize 4096 is less than device physical sector size 4194304
switching to logical sector size 512
meta-data=/dev/sdd1              isize=256    agcount=4, agsize=654848 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=2619392, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@summer163-200 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
├─sda2        8:2    0    5G  0 part /usr/summer_vfs
└─sda3        8:3    0   94G  0 part 
  └─os-root 252:0    0   94G  0 lvm  /
sdb           8:16   0   30G  0 disk 
└─sdb1        8:17   0   30G  0 part /var/lib/ceph/osd/ceph-0
sdc           8:32   0   30G  0 disk 
└─sdc1        8:33   0   30G  0 part /var/lib/ceph/osd/ceph-1
sdd           8:48   0   10G  0 disk 
└─sdd1        8:49   0   10G  0 part 
sr0          11:0    1 1024M  0 rom  
[root@summer163-200 ~]# mkdir /iscsd-dir
[root@summer163-200 ~]# mount -t xfs /dev/sdd1 /iscsd-dir/
[root@summer163-200 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.7G     0  1.7G   0% /dev
tmpfs                1.7G   20K  1.7G   1% /dev/shm
tmpfs                1.7G   17M  1.7G   1% /run
tmpfs                1.7G     0  1.7G   0% /sys/fs/cgroup
/dev/mapper/os-root   93G   38G   51G  43% /
/dev/sda2            4.8G  695M  3.9G  15% /usr/summer_vfs
/dev/sda1            976M  385M  525M  43% /boot
tmpfs                342M     0  342M   0% /run/user/1000
/dev/sdb1             30G   21G  9.9G  68% /var/lib/ceph/osd/ceph-0
tmpfs                342M     0  342M   0% /run/user/0
/dev/sdc1             30G   21G  9.9G  68% /var/lib/ceph/osd/ceph-1
/dev/sdd1             10G   33M   10G   1% /iscsd-dir
[root@summer163-200 ~]#

9. 测试写文件

[root@summer163-200 ~]# cd /iscsd-dir/
[root@summer163-200 iscsd-dir]# ll
total 0
[root@summer163-200 iscsd-dir]# mkdir xsw
[root@summer163-200 iscsd-dir]# ll
total 0
drwxr-xr-x 2 root root 6 Mar 25 10:41 xsw
[root@summer163-200 iscsd-dir]# dd if=/dev/zero of=/iscsd-dir/xsw/lalala bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.1494 s, 91.2 MB/s
[root@summer163-200 iscsd-dir]# cd xsw/
[root@summer163-200 xsw]# ll
total 102400
-rw-r--r-- 1 root root 104857600 Mar 25 10:42 lalala
[root@summer163-200 xsw]# du -sh *
100M	lalala
[root@summer163-200 xsw]#

10. 断开连接

[root@summer163-200 ~]# umount /dev/sdd1
[root@summer163-200 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.7G     0  1.7G   0% /dev
tmpfs                1.7G   20K  1.7G   1% /dev/shm
tmpfs                1.7G   17M  1.7G   1% /run
tmpfs                1.7G     0  1.7G   0% /sys/fs/cgroup
/dev/mapper/os-root   93G   38G   51G  43% /
/dev/sda2            4.8G  695M  3.9G  15% /usr/summer_vfs
/dev/sda1            976M  385M  525M  43% /boot
tmpfs                342M     0  342M   0% /run/user/1000
/dev/sdb1             30G   21G  9.9G  68% /var/lib/ceph/osd/ceph-0
tmpfs                342M     0  342M   0% /run/user/0
/dev/sdc1             30G   21G  9.9G  68% /var/lib/ceph/osd/ceph-1
[root@summer163-200 ~]# iscsiadm -m node --logoutall=all
Logging out of session [sid: 1, target: iqn.2021-03.com.60:192.168.163.sn.c0a9bbaa7884c15d55e347a7, portal: 192.168.163.201,3260]
Logout of [sid: 1, target: iqn.2021-03.com.60:192.168.163.sn.c0a9bbaa7884c15d55e347a7, portal: 192.168.163.201,3260] successful.
[root@summer163-200 ~]#

11. 删除分区

sdm      8:192  0 447.1G  0 disk 
└─sdm1   8:193  0 447.1G  0 part 
root@Kylin102:~# fdisk /dev/sdm

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


命令(输入 m 获取帮助)： d
Selected partition 1
Partition 1 has been deleted.

命令(输入 m 获取帮助)： w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

补充

> 1.发现iscsi存储: iscsiadm -m discovery -t st -p ISCSI_IP:PORT
> 2.查看iscsi发现记录 iscsiadm -m node
> 3.删除iscsi发现记录 iscsiadm -m node -o delete -T LUN_NAME -p ISCSI_IP:PORT
> 4.登录iscsi存储 iscsiadm -m node -T LUN_NAME -p ISCSI_IP:PORT -l
> 5.登出iscsi存储 iscsiadm -m node -T iqn.2017-12.com.example:storage1 -p 172.25.254.212 -l

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2023-03-29，如有侵权请联系 cloudcommunity@tencent.com 删除

存储