如果存储服务器到交换机只有一条线路的时候,那么一条线路出线故障,整个就没法使用了,所以多线路可以解决这个问题,避免单点故障。
服务端:docker-01 IP:172.17.1.150 eth0 ,br ;IP:172.17.1.160 eth1 , vmnet4
客户端:docker-02 IP:172.17.1.151 eth0 ,br ;IP:172.17.1.170 eth1 , vmnet4
服务器端:target docker-01
客户端: initiator docker-02
#172.17.1.150(docker-01)
[root@docker-01 network-scripts]# cd /etc/sysconfig/network-scripts/
[root@docker-01 network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@docker-01 network-scripts]# vim ifcfg-eth1
DEVICE=eth1 ##修改
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=172.17.1.160 ##修改
NETMASK=255.255.240.0
GATEWAY=172.17.0.1
[root@docker-01 network-scripts]# service network restart
#172.17.1.151(docker-02)
[root@docker-02 network-scripts]# cd /etc/sysconfig/network-scripts/
[root@docker-02 network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@docker-02 network-scripts]# vim ifcfg-eth1
DEVICE=eth1 ##修改
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=172.17.1.170 ##修改
NETMASK=255.255.240.0
GATEWAY=172.17.0.1
[root@docker-02 network-scripts]# service network restart
[root@docker-01 network-scripts]# yum install -y scsi-target-utils
[root@docker-01 network-scripts]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m forhelp): p
Disk /dev/sda: 107.4 GB, 107374182400bytes
255heads, 63sectors/track, 13054cylinders
Units =cylinders of 16065* 512=8225280bytes
Sector size (logical/physical): 512bytes / 512bytes
I/O size (minimum/optimal): 512bytes / 512bytes
Disk identifier: 0x000b076e
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83Linux
Partition 1does not end on cylinder boundary.
/dev/sda2 26 287 2097152 82Linux swap / Solaris
Partition 2does not end on cylinder boundary.
/dev/sda3 287 13055 102554624 83Linux
Command (m forhelp): d
Partition number (1-4): 2
Command (m forhelp): p
Disk /dev/sda: 107.4 GB, 107374182400bytes
255heads, 63sectors/track, 13054cylinders
Units =cylinders of 16065* 512=8225280bytes
Sector size (logical/physical): 512bytes / 512bytes
I/O size (minimum/optimal): 512bytes / 512bytes
Disk identifier: 0x000b076e
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83Linux
Partition 1does not end on cylinder boundary.
/dev/sda3 287 13055 102554624 83Linux
Command (m forhelp): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 4
First cylinder (26-13054, default 26):
Using default value 26
Last cylinder, +cylinders or +size{K,M,G} (26-286, default 286): +10G
Value out of range.
Last cylinder, +cylinders or +size{K,M,G} (26-286, default 286):
Using default value 286
Command (m forhelp): p
Disk /dev/sda: 107.4 GB, 107374182400bytes
255heads, 63sectors/track, 13054cylinders
Units =cylinders of 16065* 512=8225280bytes
Sector size (logical/physical): 512bytes / 512bytes
I/O size (minimum/optimal): 512bytes / 512bytes
Disk identifier: 0x000b076e
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83Linux
Partition 1does not end on cylinder boundary.
/dev/sda3 287 13055 102554624 83Linux
/dev/sda4 26 286 2091471 83Linux
Partition table entries are not indisk order
Command (m forhelp): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: 设备或资源忙.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@docker-01 network-scripts]# reboot
[root@docker-01 ~]# vim /etc/tgt/targets.conf
##在参考这段内容并在段内容后,追加以下红色标记内容:
75#<target iqn.2008-09.com.example:server.target4>
76# direct-store /dev/sdb # Becomes LUN 1
77# direct-store /dev/sdc # Becomes LUN 2
78# direct-store /dev/sdd # Becomes LUN 3
79# write-cache off
80# vendor_id MyCompany Inc.
81#</target>
##改为
82<target iqn.2019-09.cn.docker.www:target_san1>
83 backing-store /dev/sda4
84 initiator-address 172.17.1.151
85 initiator-address 172.17.1.170 #在访问控制列表中添加一个客户端
86 vendor_id docker
87 product_id target1
88</target>
##注释:
default-driver iscsi #此配置文件默认全部注释,使用iscsi驱动
<target iqn.2019-09.cn.docker.www:target_san1> # iscsi正规名字格式 : iqn.年-月.主机名倒着写: target端名字
backing-store /dev/sda4 # 可以是具体的分区,也可以是DD出来的文件。不能小于1G。(后面的文件系统是GFS,光日志空间就128M)
initiator-address 172.17.1.150 #指定允许访问的此存储主机
initiator-address 172.17.1.151 #指定允许访问的此存储主机
vendor_id “docker” #供应厂商编号 标识这个设备(字符不要过长)
product_id "target1" # 产品编号
</target>
[root@docker-01 scripts]# service tgtd restart
Stopping SCSI target daemon: not running [FAILED]
Starting SCSI target daemon: [ OK ]
[root@docker-01 scripts]# netstat -antup | grep 3260
tcp 0 00.0.0.0:3260 0.0.0.0:* LISTEN 1328/tgtd
[root@docker-01 scripts]# chkconfig tgtd on
[root@docker-01 scripts]# tgt-admin --show
Target 1: iqn.2019-09.cn.docker.www:target_san1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 2142MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda4
Backing store flags:
Account information:
ACL information:
172.17.1.151
172.17.1.170
[root@docker-02 network-scripts]# yum install -y iscsi-initiator-utils
[root@docker-02 network-scripts]# iscsiadm -m discovery -t sendtargets -p 172.17.1.150:3260
正在启动 iscsid: [确定]
172.17.1.150:3260,1 iqn.2019-09.cn.docker.www:target_san1
[root@docker-02 network-scripts]# service iscsi restart
Stopping iscsi: [ OK ]
Starting iscsi: [ OK ]
[root@docker-02 network-scripts]# ls /dev/sdb
/dev/sdb
[root@docker-02 network-scripts]# netstat -antup | grep 3260
tcp 0 0172.17.1.151:44880 172.17.1.150:3260 ESTABLISHED 2062/iscsid
[root@docker-02 network-scripts]# iscsiadm -m discovery -t sendtargets -p 172.17.1.160
172.17.1.160:3260,1 iqn.2019-09.cn.docker.www:target_san1
[root@docker-02 network-scripts]# yum install -y tree
[root@docker-02 network-scripts]# tree /var/lib/iscsi/
/var/lib/iscsi/
├── ifaces
├── isns
├── nodes
│ └── iqn.2019-09.cn.docker.www:target_san1
│ ├── 172.17.1.150,3260,1
│ │ └── default
│ └── 172.17.1.160,3260,1
│ └── default
├── send_targets
│ ├── 172.17.1.150,3260
│ │ ├── iqn.2019-09.cn.docker.www:target_san1,172.17.1.150,3260,1,default -> /var/lib/iscsi/nodes/iqn.2019-09.cn.docker.www:target_san1/172.17.1.150,3260,1
│ │ └── st_config
│ └── 172.17.1.160,3260
│ ├── iqn.2019-09.cn.docker.www:target_san1,172.17.1.160,3260,1,default -> /var/lib/iscsi/nodes/iqn.2019-09.cn.docker.www:target_san1/172.17.1.160,3260,1
│ └── st_config
├── slp
└── static
13directories, 4files
[root@docker-02 network-scripts]# service iscsi restart
Stopping iscsi: [OK]
Starting iscsi: [OK]
[root@docker-02 network-scripts]# ls /dev/sd*
/dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb /dev/sdc
[root@docker-02 network-scripts]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x1d8c9593.
Changes will remain inmemory only, untilyou decide to writethem.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m forhelp): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1022, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1022, default 1022):
Using default value 1022
Command (m forhelp): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@docker-02 network-scripts]# mkfs.ext4 /dev/sdb1
[root@docker-02 network-scripts]# mount /dev/sdb1 /opt/
[root@docker-02 network-scripts]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext4 97G 2.5G 89G 3% /
tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 ext4 190M 52M 129M 29% /boot
/dev/sdb1 ext4 2.0G 3.0M 1.9G 1% /opt
##注:可以看到sdb1可以正常使用
[root@docker-02 network-scripts]# ls /dev/sdc* ##查看不到sdc1
/dev/sdc
[root@docker-02 network-scripts]# service iscsi restart
Stopping iscsi: [OK]
Starting iscsi: [OK]
[root@docker-02 network-scripts]# ls /dev/sdc*
/dev/sdc /dev/sdc1
[root@docker-02 network-scripts]# mkdir /tmp/sdc1
[root@docker-02 network-scripts]# mount /dev/sdc1 /tmp/sdc1/
[root@docker-02 network-scripts]# mount /dev/sdb1 /opt/ ## 挂载报错了吧?
mount: you must specify the filesystem type
[root@docker-02 network-scripts]# blkid /dev/sdc1 /dev/sdb1 ##对比一下,发现UUID一样的
/dev/sdc1: UUID="d6493ee6-b9c8-4d28-9920-efc46c0354dd"TYPE="ext4"
##解决方法
[root@docker-02 network-scripts]# mount -t xfs -o nouuid /dev/sdb1 /opt
[root@docker-02 network-scripts]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext4 97G 2.5G 89G 3% /
tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 ext4 190M 52M 129M 29% /boot
/dev/sdb1 ext4 2.0G 3.0M 1.9G 1% /opt
/dev/sdc1 ext4 2.0G 3.0M 1.9G 1% /tmp/sdc1
多路径软件Device Mapper Multipath(DM-Multipath)可以将服务器节点和存储阵列之间的多条I/O链路配置为一个单独的设备。这些I/O链路是由不同的线缆、交换机、控制器组成的SAN物理链路。Multipath将这些链路聚合在一起,生成一个单独的新的设备。
(1)数据冗余
DM-Multipath可以实现在active/passive模式下的灾难转移。在active/passive模式下,只有一半的链路在工作,如果链路上的某一部分(线缆、交换机、控制器)出现故障,DM-Multipath就会切换到另一半链路上。
(2)提高性能
DM-Multipath也可以配置为active/active模式,从而I/O任务以round-robin的方式分布到所有的链路上去。通过配置,DM-Multipath还可以检测链路上的负载情况,动态地进行负载均衡。
[root@docker-02 network-scripts]# yum install device-mapper-multipath -y
[root@docker-02 network-scripts]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/
[root@docker-02 network-scripts]# service multipathd restart
ux_socket_connect: No such file or directory
Stopping multipathd daemon: [FAILED]
Starting multipathd daemon: [ OK ]
[root@docker-02 network-scripts]# multipath -ll ##没有输出消息。表示出错了。
##解决:把正在使用中的sdb1和sdbc1卸载了
[root@docker-02 network-scripts]# umount /opt/
[root@docker-02 network-scripts]# service iscsi restart
Stopping iscsi: [ OK ]
Starting iscsi: [ OK ]
[root@docker-02 network-scripts]# service multipathd restart
ok
Stopping multipathd daemon: [OK]
Starting multipathd daemon: [OK]
[root@docker-02 network-scripts]# multipath -ll
mpathb (1IET 00010001) ##(远程存储设备id) dm-0 ##(厂商)docker,target1 ##(产品ID)
size=2.0G features='0'hwhandler='0'wp=rw
|-+-policy='round-robin 0'prio=1status=active ##默认使用active的链路
|`- 8:0:0:1 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
##有效的链路,但不是活动链路
`- 7:0:0:1 sdc 8:32 active ready running
##默认配置并不会实现负载均衡,只会实现高可用的效果。
##课外讲解
00010001远程存储设备的产品信息,由于sda和sdb对应的都是远端同一个存储,所以产品信息是一样的,最后被多路经驱动发现,所以为他们生成了一个设备文件/dev/mapper/mpatha
[root@docker-02 network-scripts]# ll /dev/mapper/mpathb*
lrwxrwxrwx 1root root 79月 3022:55 /dev/mapper/mpathb -> ../dm-0
lrwxrwxrwx 1root root 79月 3022:55 /dev/mapper/mpathbp1 -> ../dm-1
[root@localhost ~]# /lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdb
1IET_00010001
[root@docker-02 network-scripts]# vim /etc/multipath.conf
##注销掉下面三行内容
25defaults {
26 user_friendly_names yes
27} ##请在此模版下写内容
28multipaths {
29 multipath {
30 wwid "1IET_00010001" #填写硬盘产品关键信息
31 alias webdata #自定义名字
32 path_grouping_policy multibus
33 path_selector "round-robin 0"
34 failback manual
35 rr_weight priorities
36 no_path_retry 5
37 }
38}
[root@docker-02 network-scripts]# service multipathd restart
ok
Stopping multipathd daemon: [OK]
Starting multipathd daemon: [OK]
[root@docker-02 network-scripts]# service iscsi restart ##需要重新实别一下存储
Stopping iscsi: [ OK ]
Starting iscsi: [ OK ]
##再次查看
[root@docker-02 network-scripts]# multipath -ll
1IET 00010001dm-0 docker,target1
size=2.0G features='0'hwhandler='0'wp=rw
|-+-policy='round-robin 0'prio=1status=active
| `- 10:0:0:1 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 9:0:0:1 sdd 8:48 active ready running
##注:之前相当于主备,现在相当于负载均衡
[root@docker-01 scripts]# ifdown eth1
[root@docker-02 mapper]# multipath -ll ##多路经驱动需要约1分钟的时间,去识别链路故障
1IET 00010001dm-0 docker,target1
size=2.0G features='0'hwhandler='0'wp=rw
|-+-policy='round-robin 0'prio=0status=active
| `- 10:0:0:1 sde 8:64 failed faulty running
##链接运行不正常 faulty故障
`-+- policy='round-robin 0' prio=1 status=enabled
`- 9:0:0:1 sdd 8:48 active ready running
##会卡在这个地方,大约一分钟后,检测故障结束
[root@docker-01 scripts]# ifup eth1
Determining ifip address 172.17.1.160 is already inuse fordevice eth1...
[root@docker-02 mapper]# multipath -ll
1IET 00010001dm-0 docker,target1
size=2.0G features='0'hwhandler='0'wp=rw
|-+-policy='round-robin 0'prio=1status=active
| `- 10:0:0:1 sde 8:64 active ready running
##链路运行正常
`-+- policy='round-robin 0' prio=1 status=enabled
`- 9:0:0:1 sdd 8:48 active ready running
udev概述:udev 是Linux kernel 2.6系列的设备管理器。它主要的功能是管理/dev目录底下的设备节点。udev会根据用户添加/删除硬件的行为,自处理/dev目录下所有设备文件。
主配置文件**:/etc/udev/udev.conf**
设备文件名字规则目录:
[root@localhost ~]# ls /lib/udev/rules.d/10-dm.rules
/lib/udev/rules.d/10-dm.rules
命名规律:开头为数字, 结尾是 .rules
运行机制和注意事项:
1、udev按照规则文件名的数字顺序来查询全部规则文件,然后为匹配规则的设备,创建其设备文件或文件链接。
2、通常情况下,建议让自己想要的规则文件最先被解析。比如,创建一个名为 /etc/udev/rules.d/10-myrule.rules的文件,并把你的规则写入该文件,这样udev就会在解析系统默认的规则文件之前解析到你的文件。
3、在规则文件里,除了以“#”开头的行(注释),所有的非空行都被视为一条规则,但是一条规则不能扩展到多行。
4、规则都是由多个键值对(key-valuepairs)组成,并由逗号隔开,键值对可以分为条件匹配键值对(以下简称“匹配键”)和赋值键值对(以下简称“赋值键”),一条规则可以有多条匹配键和多条赋值键。
5、匹配键是匹配一个设备属性的条件,当一个设备的属性匹配了该规则里所有的匹配键,就认为这条规则生效,然后按照赋值键的内容,执行该规则的赋值。
说明:如果有一个设备被内核实别sdb1,则该条件生效,执行后面的赋值:在/dev下产生一个名为my_ disk的设备文件,并把设备文件的权限设为0666。
详细参数说明
udev**规则的匹配键**
udev**重要的赋值键**
udev**规则操作符**
# KERNEL是匹配键,NAME和MODE是赋值键。
仅当操作符是“==”或者“!=”时,其为匹配键;若为其他操作符时,都是赋值键。
相关文章:http://www.361way.com/udev-disk-order/3954.html
实战:在应用服务器172,17,1,151上使用udev规则为每个target创建固定名字的软链接
对iscsi多路径共享的设备设备,让内核识别成:my_disk
在进行udev绑定的时候,可以是一个未分区的磁盘,也可以是磁盘的一个分区。
查看共享设备磁盘的UUID:
[root@localhost ~]# /lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdb
1IET_00010001
[root@localhost ~]# vim /etc/udev/rules.d/10-sda.rules
KERNEL=="sd?1",SUBSYSTEM=="block",PROGRAM="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1IET_00010001", SYMLINK+="my-disk1", MODE="0660"
参数:
# KERNEL是匹配键, SUBSYSTEM:设备的子系统名称,例如:sda 的子系统为block。udevadm info -a -p /sys/block
# PROGRAM:调用外部命令;
# RESULT:外部命令 PROGRAM 的返回结果。
# SYMLINK:为/dev/下的设备文件产生符号链接。由于udev 只能为某个设备产生一个设备文件,所以为了不覆盖系统默认的udev 规则所产生的文件,推荐使用符号链接。
# MODE:为设备设定权限
通知内核磁盘信息的变化
[root@localhost opt]# cd /dev/mapper/
[root@localhost mapper]# ls
1IET\x20\x20\x20\x20\x20000100011IET\x20\x20\x20\x20\x2000010001p1
[root@localhost mapper]# mount /dev/mapper/1IET\\x20\\x20\\x20\\x20\\x2000010001p1 /opt/
[root@localhost mapper]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext4 97G 2.6G 89G 3% /
tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 ext4 190M 52M 129M 29% /boot
/dev/mapper/1IET\x20\x20\x20\x20\x2000010001p1
ext4 2.0G 3.0M 1.9G 1% /opt