Cofiguring Multipathing

Install Multipath Package.

yum install device-mapper-multipath.x86_64

Enable The Multipathing Service.

mpathconf
chkconfig multipathd on
mpathconf --find_multipaths y
service multipathd start

List Multipath Disks

multipath -ll

Example output:

mpathc (360002ac00000000000003e9e0000af6d) dm-4 3PARdata,VV
size=2.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:1:1 sdf 8:80  active ready running
|- 1:0:0:1 sdg 8:96  active ready running
|- 2:0:1:1 sdh 8:112 active ready running
`- 2:0:0:1 sdi 8:128 active ready running

Zoning FC On Linux

Find the WWN for the Ports

cat /sys/class/fc_host/host*/port_name
ls -l /sys/class/fc_host/host*/port_name

Issuing a LIP on the HBA

echo 1 > /sys/class/fc_host/host{n}/issue_lip 
echo 1 > /sys/class/fc_host/host{n}/issue_lip

Fibre Channel Commands

A great tool for linux is systool, it does not come installed by default.
It can be installed by installing these 2 packages.

yum install libsysfs.x86_64 sysfsutils.x86_64

Disk Configuration.

About Device Names And Paths In A Multipath Configuration

There are several term for devices that are mentioned in this article. physical devices and multipath devices.

Physical Device Paths

Physical devices names and paths are what the OS sees from the array itself, and it uses those device names to group them as a single multipath device.

These names are typically named /dev/sd{X}, to determine the physical device names that are used for a multipath device, you can run the following command:

multipath -l

You should get an output similar to this:

mpathb (360002ac00000000000003af40000af6b) dm-3 3PARdata,VV
size=8.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 1:0:1:0 sdc 8:32 active undef running
|- 1:0:0:0 sdb 8:16 active undef running
|- 2:0:0:0 sde 8:64 active undef running
`- 2:0:1:0 sdd 8:48 active undef running

In this example, the physical device names are sdb, sdc, sdd, and sde.

Multipath Device Paths

Multipath device paths are an aggregate block device that device-mapper creates from the physical devices. The devices names typically start with mpath followed by a letter. they are located in /dev/mapper/. These names are used to create file-systems etc. on the device.

You can determine the name by running the following command:

multipath -l

In this case, the Multipath Device name is mpathb, and it would be located at /dev/mapper/mpathb

LVM and Multipathing software

By default LVM scans all block devices under /dev, including /dev/sdb etc, in which the case may be that they are part of a multipath device, therefore causing error to be thrown when running LVM commands like lvs or lvchange.

The workaround to this, is to filter out all devices that start with /dev/sd*, except the device on which the OS is running. To accomplish this, add a filter to /etc/lvm/lvm.conf as follows.

filter = [ "a|/dev/sda2$|", "a|/dev/mapper/mpath.*|", "r|.*|" ]

To test if the syntax does what you want it to do, you can run:

pvs --config 'devices{ filter = [ "a|/dev/sda2$|", "a|/dev/mapper/mpath.*|", "r|.*|" ] }'

Managing Volumes

Creating A Volume Group

vgcreate hot /dev/mapper/mpathc

Creating a Logical Volume

This will create a logical volume on the entire disk.

lvcreate -l 100%VG disk1

Creating a Filesystem On The Logical Volume

mkfs.ext4 /dev/disk1/lvol0

Resizing Volumes

Find The physical device name for every path.

multipath -ll
mpathb (360002ac00000000000002ae20000af6b) dm-3 3PARdata,VV
size=5.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
 |- 1:0:0:1 sdc 8:32  active ready running
 |- 1:0:1:1 sde 8:64  active ready running
 |- 2:0:0:1 sdg 8:96  active ready running
 `- 2:0:1:1 sdi 8:128 active ready running

Rescan each individual device path.

Write a 1 to the rescan file for the device

echo 1 > /sys/block/sd{X}/device/rescan

Resize your device by running the multipathd resize command for the multipath device.

multipathd -k'resize map mpath{X}'

Confirm the new device size

multipath -ll
mpathb (360002ac00000000000002ae20000af6b) dm-3 3PARdata,VV
size=8.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
 |- 1:0:0:1 sdc 8:32  active ready running
 |- 1:0:1:1 sde 8:64  active ready running
 |- 2:0:0:1 sdg 8:96  active ready running
 `- 2:0:1:1 sdi 8:128 active ready running

Find the physical device path.

pvdisplay
--- Physical volume ---
PV Name               /dev/mapper/mpathb
VG Name               cold
PV Size               5.00 TiB / not usable 16.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              2097148
Free PE               0
Allocated PE          2097148
PV UUID               ckE1fc-G4YC-2cl9-H1LN-GVWT-QyKi-xZxWXg

Resize the Physical volume.

pvresize -v /dev/mapper/mpathb
Using physical volume(s) on command line
Archiving volume group "cold" metadata (seqno 2).
Resizing volume "/dev/mapper/mpathb" to 10737418240 sectors.
Resizing physical volume /dev/mapper/mpathb from 0 to 2097148 extents.
Updating physical volume "/dev/mapper/mpathb"
Creating volume group backup "/etc/lvm/backup/cold" (seqno 3).
Physical volume "/dev/mapper/mpathb" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

Check the Physical Volume if it has changed.

pvdisplay /dev/mapper/mpathb
--- Physical volume ---
PV Name               /dev/mapper/mpathb
VG Name               cold
PV Size               8.00 TiB / not usable 16.00 MiB
Allocatable           yes
PE Size               4.00 MiB
Total PE              2097148
Free PE               786432
Allocated PE          1310716
PV UUID               ckE1fc-G4YC-2cl9-H1LN-GVWT-QyKi-xZxWXg

You can see the size has increased to 8 TB

Resize the Logical volume.

lvextend -l +100%FREE /dev/mapper/cold-lvol0

Check if the size of the Logical Volume has indeed increased.

lvdisplay /dev/cold/lvol0
--- Logical volume ---
LV Path                /dev/cold/lvol0
LV Name                lvol0
VG Name                cold
LV UUID                4vZQrw-3Gzo-6EyZ-dgcc-uD7L-K7IF-qlykMw
LV Write Access        read/write
LV Creation host, time spix4, 2014-06-10 17:31:30 -0400
LV Status              available
# open                 1
LV Size                8.00 TiB
Current LE             2097148
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:4

Check Filesystem for erroros.

Before you do fsck, you need to unmount the drive.

e2fsck -f /dev/mapper/cold-lvol0

Resize the Partition.

resize2fs /dev/mapper/cold-lvol0

Remove Multipath Device

multipath -f {devid}

For {devid} use the device name from the output on multipath -l. e.g.

multipath -f mpathb

Vendor Specific Parameters.

For 3PAR

Add these lines to the end of /etc/multipath.conf in the devices section.

	device {
vendor "3PARdata"
product "VV"
no_path_retry 18
features "0"
hardware_handler "0"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_selector "round-robin 0"
rr_weight uniform
rr_min_io_rq 1
path_checker tur
failback immediate
}

For Violin

  device {
vendor                "VIOLIN"
product               "SAN ARRAY"
getuid_callout        "/sbin/scsi_id --whitelisted --replace-whitespace --page=0x80 --device=/dev/%n"
path_grouping_policy  group_by_serial
path_checker          tur
path_selector         "round-robin 0"
hardware_handler      "0"
fast_io_fail_tmo      5
dev_loss_tmo          30
failback              immediate
features              "0"
rr_weight             uniform
no_path_retry         fail
rr_min_io_rq          4
}

For Hitachi VSP

	device {
vendor			"HITACHI"
product			"OPEN.*"
path_grouping_policy	multibus
path_checker		tur
no_path_retry		1
}

For Kaminario

	device {
vendor			"KMNRIO"
product			"K2"
path_grouping_policy	multibus
getuid_callout		"/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_checker		directio
path_selector		"queue-length 0"
no_path_retry		fail
hardware_handler	"0"
rr_weight		priorities
rr_min_io		1
failback		15
fast_io_fail_tmo	5
dev_loss_tmo		8
}

For Pure Flash Array

Create the udev file fop pure-storage and add the following content to it.

vi /etc/udev/rules.d/99-pure-storage.rules

# Recommended settings for Pure Storage FlashArray.

# Use noop scheduler for high-performance solid-state storage
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="noop"

# Reduce CPU overhead due to entropy collection
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"

# Spread CPU load by redirecting completions to originating CPU
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"

# Set the HBA timeout to 60 seconds
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{model}=="FlashArray      ", RUN+="/bin/sh -c 'echo 60 > /sys$DEVPATH/device/timeout'"

Add the following line to the devices section in /etc/multipath.conf.

   device {
vendor                "PURE"
path_selector         "round-robin 0"
path_grouping_policy  multibus
path_checker          tur
fast_io_fail_tmo      10
dev_loss_tmo          60
no_path_retry         0
}