SOLVED DS4243 LSI 9300-8e Multipathing

judokan9

Dabbler
Joined
Sep 26, 2018
Messages
13
Hello xnaron,

yes currently i am running two DS4246 with two IOM6 controllers each. Every enclosure is hooked up to two servers with CentOS7, zfs-on-linux and gluster on there. So far everything works like a charm, i have nothing to complain about. All my drives are negotiating directly to the right speeds even after serveral reboots and power cycles.

I never tried SATA drives because i want two servers connect to each dirve for high avalibility.

Sadly we had a few power outages the last days and one drive did not spunn up any more.
I orderet a new replacement drive (10000GB Seagate Exos X10 ST10000NM0096 256MB) this thing also works great in the enclosure.

Yeah those cables are freaking expensive.... about 80€ (91,12$) for new cable. And i needed eight of them because of the high avalibility.

Performance is also very good. I suggest you to upgrade the LSI cards to the newest IT-firmware (two links who helped me to do that LINK1 LINK2) because after this i got like 300-500 MB/s more in write and read is now about 4,5-5 GB/s also a few MB/s better than bevore.

I hope i helped you a bit.
If you have any other questions feel free to ask, i did a lot of stuff with those DS4243 the last weeks.
 

xnaron

Explorer
Joined
Dec 11, 2014
Messages
98
That sucks about the power outages. I've got a ups that can provide about 40min to my rack and gives me enough time to shut things down. I have some of it scripted.

Thanks for the links and help. I'll definitely be upgrading the firmware when my 9300-8e arrives. It will be interesting to see if that changes anything with my sata drive speed negotiation with the 9300. I doubt it will but at this point I'm running out of things to try.
 

judokan9

Dabbler
Joined
Sep 26, 2018
Messages
13
Yeah, the UPS is on it's way to me :)

No problem, i hope your problems can be solved with the new controller. I can't find the native block size of your Seagate drives in the datasheed, so can you tell me how big it is?
 

xnaron

Explorer
Joined
Dec 11, 2014
Messages
98
Good. The UPS will give you piece of mind.

They are 4096 byte sectors. FreeNAS is using ashift with them.
 

judokan9

Dabbler
Joined
Sep 26, 2018
Messages
13
Okay, that is odd. I know that the IOM's and have problems or don't even work with drives that have a BS larger then 512 bytes. Do your drives have a 512 byte emulation?

Saldy i don't know anything about the firmware of the IOM's, but when you know how to upgrade them etc. let me know.
 

xnaron

Explorer
Joined
Dec 11, 2014
Messages
98
Okay, that is odd. I know that the IOM's and have problems or don't even work with drives that have a BS larger then 512 bytes. Do your drives have a 512 byte emulation?

Saldy i don't know anything about the firmware of the IOM's, but when you know how to upgrade them etc. let me know.

Yes I think the drives have 512 byte emulation. In order to upgrade the firmware of the IOM6 you need to do it through a NetAPP device. I think a licence is required.
 

sunshine931

Explorer
Joined
Jan 23, 2018
Messages
54
I'm also running a DS4246 and looking for guidance around optimal cabling and gmultipath config to achieve dual-link connections to the SAS devices. I have 24 SAS drives installed and my hope is to achieve better throughput, iops, and resiliency.

I've been reading the examples of camcontrol smpphylist /dev/sesX output.

In my case, I'm cabled from my two-port HBA to the DS4246's IOMs using the square port on each IOM. I see TWO enclosures. One via each connection I gather.

I see 4 connections that I believe to be my HBA, visible to each enclosure as a different phy (0x5b083fe0d8d9fe00 and 0x5b083fe0d8d9fe01)

For others using the DS4246, I'm curious about -
  1. Do you use multiple IOM6s?
  2. Do you see multiple SES devices?
  3. How did you cable your HBA to the IOM6s?

First, from ses0:
Code:
[root@omega /usr/local]# camcontrol smpphylist /dev/ses0
37 PHYs:
PHY  Attached SAS Address
  0  0x5b083fe0d8d9fe00
  1  0x5b083fe0d8d9fe00
  2  0x5b083fe0d8d9fe00
  3  0x5b083fe0d8d9fe00
  4  0x0000000000000000
  5  0x0000000000000000
  6  0x0000000000000000
  7  0x0000000000000000
  8  0x5000cca01a9fdbcd   <HITACHI HUS72303CLAR3000 C1D6>   (pass16,da15)
  9  0x5000cca01adb9dc9   <HITACHI HUS72303CLAR3000 C1D6>   (pass5,da4)
10  0x5000cca03e14a42d   <HITACHI HUS72303CLAR3000 C1D6>   (pass6,da5)
11  0x5000cca03e6445fd   <HITACHI HUS72303CLAR3000 C1D6>   (pass7,da6)
12  0x5000cca03eb0b8f9   <HITACHI HUS723030ALS640 A222>    (pass47,da44)
13  0x5000cca03eaf8fdd   <HITACHI HUS723030ALS640 A222>    (pass48,da45)
14  0x5000cca03eaf8fcd   <HITACHI HUS723030ALS640 A222>    (pass49,da46)
15  0x5000cca03eb57f15   <HITACHI HUS723030ALS640 A222>    (pass50,da47)
16  0x5000cca046093f49   <HITACHI HUS72303CLAR3000 C442>   (pass8,da7)
17  0x5000cca0460a0e11   <HITACHI HUS72303CLAR3000 C442>   (pass9,da8)
18  0x5000cca0460a0e25   <HITACHI HUS72303CLAR3000 C442>   (pass10,da9)
19  0x5000cca03e1986b9   <HITACHI HUS72303CLAR3000 C1D6>   (pass11,da10)
20  0x5000cca03e1c4301   <HITACHI HUS72303CLAR3000 C1D6>   (pass12,da11)
21  0x5000cca0460964e5   <HITACHI HUS72303CLAR3000 C442>   (pass13,da12)
22  0x5000cca01a8a0ac9   <HITACHI HUS72303CLAR3000 C1D6>   (pass14,da13)
23  0x5000cca04609fa79   <HITACHI HUS72303CLAR3000 C442>   (pass15,da14)
24  0x5000cca03ebdd37d   <HITACHI HUS723030ALS640 A222>    (pass51,da48)
25  0x5000cca03eb631f9   <HITACHI HUS723030ALS640 A222>    (pass52,da49)
26  0x5000cca03eb0bacd   <HITACHI HUS723030ALS640 A222>    (pass53,da50)
27  0x5000cca03eb60e85   <HITACHI HUS723030ALS640 A222>    (pass54,da51)
28  0x5000cca03e562df5   <HITACHI HUS72303CLAR3000 C1D6>   (pass17,da16)
29  0x5000cca03e008515   <HITACHI HUS72303CLAR3000 C1D6>   (pass18,da17)
30  0x5000cca01aa14d25   <HITACHI HUS72303CLAR3000 C1D6>   (pass19,da18)
31  0x5000cca03e269129   <HITACHI HUS72303CLAR3000 C1D6>   (pass20,da19)
32  0x0000000000000000
33  0x0000000000000000
34  0x0000000000000000
35  0x0000000000000000
36  0x500a098004347dbe
[root@omega /usr/local]#


And from ses1:
Code:
[root@omega /usr/local]# camcontrol smpphylist /dev/ses1
37 PHYs:
PHY  Attached SAS Address
  0  0x5b083fe0d8d9fe01
  1  0x5b083fe0d8d9fe01
  2  0x5b083fe0d8d9fe01
  3  0x5b083fe0d8d9fe01
  4  0x0000000000000000
  5  0x0000000000000000
  6  0x0000000000000000
  7  0x0000000000000000
  8  0x5000cca01a9fdbce   <HITACHI HUS72303CLAR3000 C1D6>   (da20,pass22)
  9  0x5000cca01adb9dca   <HITACHI HUS72303CLAR3000 C1D6>   (da23,pass25)
10  0x5000cca03e14a42e   <HITACHI HUS72303CLAR3000 C1D6>   (da22,pass24)
11  0x5000cca03e6445fe   <HITACHI HUS72303CLAR3000 C1D6>   (da21,pass23)
12  0x5000cca03eb0b8fa   <HITACHI HUS723030ALS640 A222>    (da24,pass26)
13  0x5000cca03eaf8fde   <HITACHI HUS723030ALS640 A222>    (da25,pass27)
14  0x5000cca03eaf8fce   <HITACHI HUS723030ALS640 A222>    (da27,pass29)
15  0x5000cca03eb57f16   <HITACHI HUS723030ALS640 A222>    (da26,pass28)
16  0x5000cca046093f4a   <HITACHI HUS72303CLAR3000 C442>   (da43,pass46)
17  0x5000cca0460a0e12   <HITACHI HUS72303CLAR3000 C442>   (da30,pass32)
18  0x5000cca0460a0e26   <HITACHI HUS72303CLAR3000 C442>   (da31,pass33)
19  0x5000cca03e1986ba   <HITACHI HUS72303CLAR3000 C1D6>   (da29,pass31)
20  0x5000cca03e1c4302   <HITACHI HUS72303CLAR3000 C1D6>   (da28,pass30)
21  0x5000cca0460964e6   <HITACHI HUS72303CLAR3000 C442>   (da32,pass34)
22  0x5000cca01a8a0aca   <HITACHI HUS72303CLAR3000 C1D6>   (da33,pass35)
23  0x5000cca04609fa7a   <HITACHI HUS72303CLAR3000 C442>   (da42,pass44)
24  0x5000cca03ebdd37e   <HITACHI HUS723030ALS640 A222>    (da35,pass37)
25  0x5000cca03eb631fa   <HITACHI HUS723030ALS640 A222>    (da34,pass36)
26  0x5000cca03eb0bace   <HITACHI HUS723030ALS640 A222>    (da36,pass38)
27  0x5000cca03eb60e86   <HITACHI HUS723030ALS640 A222>    (da37,pass39)
28  0x5000cca03e562df6   <HITACHI HUS72303CLAR3000 C1D6>   (da39,pass41)
29  0x5000cca03e008516   <HITACHI HUS72303CLAR3000 C1D6>   (da38,pass40)
30  0x5000cca01aa14d26   <HITACHI HUS72303CLAR3000 C1D6>   (da40,pass42)
31  0x5000cca03e26912a   <HITACHI HUS72303CLAR3000 C1D6>   (da41,pass43)
32  0x0000000000000000
33  0x0000000000000000
34  0x0000000000000000
35  0x0000000000000000
36  0x500a098004346bfe
[root@omega /usr/local]#
 
Last edited:

sunshine931

Explorer
Joined
Jan 23, 2018
Messages
54
OK I've done some more toying around, and:

  • When both HBA connections are made to a single IOM6, I see 8 HBA connections in my camcontrol smpphylist /dev/ses0 output. Also, I only have one ses - ses0.
  • When my HBA is connected to two IOM6s, using either both square, both circle, or one of each, I see 4 HBA connection to one ses device, and 4 HBA connections to another ses device. Drives are visible with different phy's from each ses.

I've completed some benchmarks using two IOM6s and a single 2-port HBA. I'm working on the same for a single IOM6 with two connections from my HBA. Will edit with an update soon.

Here are benchmark results where the HBA sees two paths to the same devices (i.e. multipath, and I use gmultipath to set different profiles for active/active vs. active/passive vs. active/read):

for DISK in {1..24}; do gmultipath configure -A disk$DISK; done
1549047702543.png


for DISK in {1..24}; do gmultipath configure -P disk$DISK; done
1549047666221.png


for DISK in {1..24}; do gmultipath configure -R disk$DISK; done
1549047781415.png


Benchmarks for the 2-port HBA connected to a single IOM6 (i.e. dual-link, so there's no multipath to worry about configuring as there's only one path):
1549057956010.png
 
Last edited:

KenwoodFox

Explorer
Joined
Nov 6, 2018
Messages
74
I have a similar NetApp setup, but I want to configure multipath for redundancy and i don't see the WebGUI button, I still want the original question answered. If i plug in my second controller, every device shows up twice.
 
Top