After some dubious network configurations (that I should have never configured incorrectly) I finally got multipath working to the main storage server. All of the multipath.conf examples I saw resulted in non-functional iSCSI MPIO, while having no multipath.conf left me with failover MPIO instead of interleaved/round-robin.
A large issue with trying to get MPIO configured was the fact that all the examples I found were either old (and scsi_id works slightly differently in Ubuntu 14.04) or just poor. Yes, I wound up using Ubuntu. Usually I use Slackware for EVERYTHING, but lately I’ve been trying to branch out. Most of the VMs run Fedora, “Pegasus” or VMSrv1 uses Fedora, “Titan” uses Ubuntu.
Before I did anything with multipath.conf (It’s empty on Ubuntu 14.04), I got this:
root@titan:/home/frankd# multipath -ll 1FREEBSD HTPC1-D1 dm-2 FREEBSD,CTLDISK size=256G features='0' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active | `- 13:0:0:0 sde 8:64 active ready running `-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:0 sdd 8:48 active ready running
Note the disks are both round-robin — with only one member each! This works for fail-over, but did nothing for performance. The only thing that wound up working for multipath.conf was this:
defaults {
user_friendly_names yes
polling_interval 3
path_grouping_policy multibus
path_checker readsector0
path_selector "round-robin 0"
features "0"
no_path_retry 1
rr_min_io 100
}
multipaths {
multipath {
wwid 1FREEBSD_HTPC1-D1
alias testLun
}
}
The wwid/alias doesn’t work, however. All of the MPIO is just coming from the defaults stanza. I attempted many things with no luck, unfortunately. I’m going to have to delve into this more especially if I want live migrations to work properly with MPIO. As it stands the disk devices are pointing at a single IP (ex /dev/disk/by-path/ip-172.17.2.2:3260-iscsi-iqn.2014-12.lab.frankd:htpc1-lun-0), I’ll need to point at aliases to get the VMs working with multipath.
The multipath tests themselves were promising though, dd was able to give me a whopping 230MB/s to the mapper device over a pair of GigE connections.
The output from ‘multipath -ll’ now looked more reasonable:
root@titan:/home/frankd# multipath -ll mpath1 (1FREEBSD HTPC1-D1) dm-2 FREEBSD,CTLDISK size=256G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 39:0:0:0 sde 8:64 active ready running `- 40:0:0:0 sdg 8:96 active ready running
You can see the drives are both under the same round-robin policy instead of two separate ones.
The storage server also saw some slight changes, including upgrading from one Intel X25-V 40GB for L2ARC to 2xX25-Vs for a total of 80GB. I also added a 60GB Vertex 2 as a SZIL device. I really need to build a machine with more RAM and partition out the SZIL. I’ll likely wind up using my 840Pro 256GB for L2ARC and leave the old X25Vs out of the main array once I get a pair of 10GbE cards for maximum speed (hopefully near-native of the 840Pro — perhaps better with a large amount of ARC) to my workstation.
So we’re at a point where everything appears to be working, although in need of some upgrades! Great! I’m looking at a KCMA-D8 Dual Opteron C32 motherboard as I have a pair of Opteron 4184s (6 core Lisbon, very similar to a Phenom II X6 1055T) laying around, so I could put together a 32GB 12 core machine for under $400 — but as always, budgetary constraints for a hobby squash that idea quickly.