SOLVED DS4243 LSI 9300-8e Multipathing

[judokan9]

Hello,

i have an DS4243 with two IOM6 modules installed.
The diskshelf is connected to an LSI 9300-8e where i today upgraded the firmware to 16.00.01.00 via DOS.
My second IOM6 module arrived today too but i can't get multipathing up and running.

The camcontrol devlist just shows the 24 disks of the 24 installed in the shelf.

Both IOM6 modules are connected to the LSI 9300-8e.

I also don't have a "View Multipath" button in the Web-GUI

Both IOM6 modules are working and both ports on the 9300-8e functioning well.

My goal is to create an Active/Active connection to split the load between the IOM6 modules and get higher read and write speeds.

Show : camcontrol devlist

<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 8 lun 0 (da2,pass3)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 9 lun 0 (da3,pass4)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 10 lun 0 (da4,pass5)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 11 lun 0 (da5,pass6)
<NETAPP X412_S15K7560A15 NA06> at scbus3 target 12 lun 0 (pass7,da6)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 13 lun 0 (pass8,da7)
<NETAPP X412_S15K7560A15 NA08> at scbus3 target 14 lun 0 (pass9,da8)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 15 lun 0 (pass10,da9)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 16 lun 0 (pass11,da10)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 17 lun 0 (pass12,da11)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 18 lun 0 (pass13,da12)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 19 lun 0 (pass14,da13)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 20 lun 0 (pass15,da14)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 21 lun 0 (pass16,da15)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 22 lun 0 (pass17,da16)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 23 lun 0 (pass18,da17)
<NETAPP X412_S15K7560A15 NA06> at scbus3 target 24 lun 0 (pass19,da18)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 25 lun 0 (pass20,da19)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 26 lun 0 (pass21,da20)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 27 lun 0 (pass22,da21)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 28 lun 0 (pass23,da22)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 29 lun 0 (pass24,da23)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 30 lun 0 (pass25,da24)
<NETAPP X412_HVIPC560A15 NA03> at scbus3 target 31 lun 0 (pass26,da25)
<NETAPP DS424IOM6 0180> at scbus3 target 36 lun 0 (ses0,pass27)

Show : dmesg | grep mpr

mpr0: <Avago Technologies (LSI) SAS3008> port 0xc000-0xc0ff mem 0xfc990000-0xfc99ffff,0xfc940000-0xfc97ffff irq 10 at device 6.0 on pci0
mpr0: IOC in fault state 0x0
mpr0: Firmware: 16.00.01.00, Driver: 18.03.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a48e
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad948f2
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad964a2
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad9b776
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000c5006800752e
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad192ce
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000c5007156f1a6
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad193d6
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ac7a
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad93e9a
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a326
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 10
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad193fe
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 11
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1acaa
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0016> enclosureHandle<0x0002> slot 12
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ecb6
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0017> enclosureHandle<0x0002> slot 13
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ab8e
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0018> enclosureHandle<0x0002> slot 14
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ae32
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0019> enclosureHandle<0x0002> slot 15
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000c5007e101026
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001a> enclosureHandle<0x0002> slot 16
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a3ce
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001b> enclosureHandle<0x0002> slot 17
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad19b6e
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001c> enclosureHandle<0x0002> slot 18
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1adc2
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001d> enclosureHandle<0x0002> slot 19
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1eaa6
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001e> enclosureHandle<0x0002> slot 20
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02acfc866
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x001f> enclosureHandle<0x0002> slot 21
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad3c736
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0020> enclosureHandle<0x0002> slot 22
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad195ba
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0021> enclosureHandle<0x0002> slot 23
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 500a09800205a2fe
mpr0: Found device <4441<SspInit,SspTarg,SepDev>,End Device> <3.0Gbps> handle<0x0022> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name ( )
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a48d
mpr0: SAS Address from SAS device page0 = 5000cca02ad948f1
mpr0: SAS Address from SAS device page0 = 5000cca02ad964a1
mpr0: SAS Address from SAS device page0 = 5000cca02ad9b775
mpr0: SAS Address from SAS device page0 = 5000c5006800752d
mpr0: SAS Address from SAS device page0 = 5000cca02ad192cd
mpr0: SAS Address from SAS device page0 = 5000c5007156f1a5
mpr0: SAS Address from SAS device page0 = 5000cca02ad193d5
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ac79
mpr0: SAS Address from SAS device page0 = 5000cca02ad93e99
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a325
mpr0: SAS Address from SAS device page0 = 5000cca02ad193fd
mpr0: SAS Address from SAS device page0 = 5000cca02ad1aca9
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ecb5
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ab8d
mpr0: SAS Address from SAS device page0 = 5000cca02ad1ae31
mpr0: SAS Address from SAS device page0 = 5000c5007e101025
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a3cd
mpr0: SAS Address from SAS device page0 = 5000cca02ad19b6d
mpr0: SAS Address from SAS device page0 = 5000cca02ad1adc1
mpr0: SAS Address from SAS device page0 = 5000cca02ad1eaa5
mpr0: SAS Address from SAS device page0 = 5000cca02acfc865
mpr0: SAS Address from SAS device page0 = 5000cca02ad3c735
mpr0: SAS Address from SAS device page0 = 5000cca02ad195b9
mpr0: SAS Address from SAS device page0 = 500a098001b440fe
da2 at mpr0 bus 0 scbus3 target 8 lun 0
da3 at mpr0 bus 0 scbus3 target 9 lun 0
da4 at mpr0 bus 0 scbus3 target 10 lun 0
da5 at mpr0 bus 0 scbus3 target 11 lun 0
da7 at mpr0 bus 0 scbus3 target 13 lun 0
da9 at mpr0 bus 0 scbus3 target 15 lun 0
da8 at mpr0 bus 0 scbus3 target 14 lun 0
da10 at mpr0 bus 0 scbus3 target 16 lun 0
da6 at mpr0 bus 0 scbus3 target 12 lun 0
da11 at mpr0 bus 0 scbus3 target 17 lun 0
da12 at mpr0 bus 0 scbus3 target 18 lun 0
da13 at mpr0 bus 0 scbus3 target 19 lun 0
da14 at mpr0 bus 0 scbus3 target 20 lun 0
da15 at mpr0 bus 0 scbus3 target 21 lun 0
da16 at mpr0 bus 0 scbus3 target 22 lun 0
da17 at mpr0 bus 0 scbus3 target 23 lun 0
da18 at mpr0 bus 0 scbus3 target 24 lun 0
da19 at mpr0 bus 0 scbus3 target 25 lun 0
da20 at mpr0 bus 0 scbus3 target 26 lun 0
da21 at mpr0 bus 0 scbus3 target 27 lun 0
da22 at mpr0 bus 0 scbus3 target 28 lun 0
da23 at mpr0 bus 0 scbus3 target 29 lun 0
da24 at mpr0 bus 0 scbus3 target 30 lun 0
da25 at mpr0 bus 0 scbus3 target 31 lun 0
ses0 at mpr0 bus 0 scbus3 target 36 lun 0
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: mprsas_prepare_remove: Sending reset for target ID 8
da2 at mpr0 bus 0 scbus3 target 8 lun 0
mpr0: da2: clearing target 8 handle 0x000a
mpr0: <NETAPP X412_HVIPC560A15 NA03> s/n LXYR95LN detached
mpr0: (da2:Unfreezing devq for target ID 8
mpr0:0:8:0): Periph destroyed
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: mprsas_prepare_remove: Sending reset for target ID 9
da3 at mpr0 bus 0 scbus3 target 9 lun 0
mpr0: da3: <NETAPP X412_HVIPC560A15 NA03> s/n LXYVHGLN detached
(da3:mpr0:0:mpr0: 9:0): Periph destroyed
mpr0: Unfreezing devq for target ID 9
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: mprsas_prepare_remove: Sending reset for target ID 10
da4 at mpr0 bus 0 scbus3 target 10 lun 0
mpr0: da4: clearing target 10 handle 0x000c
mpr0: <NETAPP X412_HVIPC560A15 NA03> s/n LXYVK9SN detached
(da4:mpr0: mpr0:0:Unfreezing devq for target ID 10
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: _mapping_inc_missing_count: device is already removed from mapping table
mpr0: mprsas_prepare_remove: Sending reset for target ID 11
da5 at mpr0 bus 0 scbus3 target 11 lun 0
mpr0: da5: clearing target 11 handle 0x000d
mpr0: <NETAPP X412_HVIPC560A15 NA03> s/n LXYVRUSN detached
(da5:mpr0: Unfreezing devq for target ID 11
mpr0:0:11:0): Periph destroyed
mpr0: SAS Address from SAS device page0 = 5000cca02ad1a48e
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name ( )
(probe0:mpr0:0:8:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 116 terminated ioc 804b loginfo 31110e05 scsi 0 state c xfer 0
(probe0:mpr0:0:8:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:8:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:8:0): Retrying command
mpr0: SAS Address from SAS device page0 = 5000cca02ad948f2
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr0: At enclosure level 0 and connector name ( )
(probe1:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 149 terminated ioc 804b loginfo 31110e05 scsi 0 state c xfer 0
(probe1:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe1:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe1:mpr0:0:9:0): Retrying command
mpr0: SAS Address from SAS device page0 = 5000cca02ad964a2
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr0: At enclosure level 0 and connector name ( )
(probe2:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 158 terminated ioc 804b loginfo 31110e05 scsi 0 state c xfer 0
(probe2:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe2:mpr0:0:10:0): CAM status: CCB request completed with an error
(probe2:mpr0:0:10:0): Retrying command
mpr0: SAS Address from SAS device page0 = 5000cca02ad9b776
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr0: At enclosure level 0 and connector name ( )
(probe3:mpr0:0:11:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 173 terminated ioc 804b loginfo 31110e05 scsi 0 state c xfer 0
(probe3:mpr0:0:11:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mpr0:0:11:0): CAM status: CCB request completed with an error
(probe3:mpr0:0:11:0): Retrying command
da2 at mpr0 bus 0 scbus3 target 8 lun 0
da3 at mpr0 bus 0 scbus3 target 9 lun 0
da4 at mpr0 bus 0 scbus3 target 10 lun 0
da5 at mpr0 bus 0 scbus3 target 11 lun 0

 

[Chris Moore]

What kind of disks do you have installed?

PS. I have only seen multi path setup using two controllers, not with both paths going back to a single controller.

 

[judokan9]

I am using NetApp SAS disks (Model Number X412_HVIPC560A15).
After reformating the blocksize to 512bytes they worked fine.

 

[kdragon75]

You won't see any performance gains even if the shelf is full of those drives. SAS multipathing is more for redundancy than anything else.

 

[judokan9]

Ohh okay,

I thought I need multipathing to split the load.
I am not that into storage servers and SAS aswell.

So how can i get more bandwidth to my drives?
There are currently 4 vdev’s (RAIDZ2) with 6 drives each striped into a „big“ volume (for testing). I get around 1,78 GB/s with a simple dd command which creates a 100GB file from /dev/zero, on a single SAS 6G connection to the shelf.

 

[Chris Moore]

You want dual link. I don't know if the enclosure supports it.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[kdragon75]

How fast is each drive? 180MB/s sequential reads ballpark. So 180x24=4320MB/s from the drives and the shelf has 4x6gb/s SAS lanes. So 4x6000=24000mb/s=3000MB/s and this is all under PERFECT conditions. I wont help performance but will keep things running is a IO module fails, cable gets pulled, etc...

 

[Johnnie Black]

So 4x6000=24000mb/s=3000MB/s and this is all under PERFECT conditions

Don't forget about 8b/10b encoding 24000Mb/s = 2400MB/s, of those around 2200MB/s are the max "real word" speed you can get due to protocol overhead, if the enclosure supports dual link it will double that.

 

[judokan9]

Ahh okay,

thank you for your explanation kdragon75 and Johnnie Black.

I am currently reading thrugh the SAS standards and learning allot.

So is there a way to check the dual link capability other than read thrugh NetApp documentations?

And lets say, I install a few SAS SSD‘s into that shelf and it‘s supporting dual link. Is it easy as connecting the second cable, the SAS magic happends and i get more bandwidth?

 

[Johnnie Black]

Is it easy as connecting the second cable, the SAS magic happends and i get more bandwidth?

If supported yes, you just need to connect the second cable.

You should be able to see if it's using single or dual link by using camcontrol smpphylist /dev/ses0

You might need to replace ses0 with the correct number, you'll get a result similar to this:

Code:

26 PHYs:
PHY  Attached SAS Address
  0  0x0000000000000000
  1  0x0000000000000000
  2  0x50080e53c2b8f002   <ATA ST1000DM003-1ER1 CC45>	   (da33,pass36)
  3  0x5000cca01ab1a139   <IBM-ESXS HUS723030ALS64 J210>	(pass0,da0)
  4  0x0000000000000000
  5  0x0000000000000000
  6  0x0000000000000000
  7  0x5000c50041affc01   <IBM-ESXS ST33000650SS BC36>	  (pass2,da2)
  8  0x0000000000000000
  9  0x0000000000000000
 10  0x5000cca03ea41585   <IBM-ESXS HUS723030ALS64 J3K7>	(pass1,da1)
 11  0x0000000000000000
 12  0x500605b004f24f20
 13  0x500605b004f24f20
 14  0x500605b004f24f20
 15  0x500605b004f24f20
 16  0x0000000000000000
 17  0x0000000000000000
 18  0x0000000000000000
 19  0x0000000000000000
 20  0x0000000000000000
 21  0x0000000000000000
 22  0x0000000000000000
 23  0x0000000000000000
 24  0x50080e53c2b8f03d
 25  0x000000000000003e

Note the 4 PHYs with the same SAS Address, PHYs 12 to 15, this is the link to the HBA, if it was using dual link you would see 8 PHYs with the same SAS address, usually 4 + 4 and both links not necessarily together, e.g., the other 4 could be PHYs 20 to 23.

 

[Johnnie Black]

Forgot I had this screenshot, better example, showing dual link:

 

[judokan9]

Thank you again i will try this tomorrow (in a few hours).

I will post the results then.

 

[kdragon75]

I install a few SAS SSD‘s into that shelf and it‘s supporting dual link. Is it easy as connecting the second cable, the SAS magic happends and i get more bandwidth?

Assuming there dual port SAS drive but most if not all enterprise SAS SSDs should be.

 

[Johnnie Black]

You don't need dual port devices for dual link, dual port is for redundancy, dual link is for increased bandwidth and it's just the link between the HBA and the expander, then any devices can use the extra bandwidth, including SATA.

 

[kdragon75]

You don't need dual port devices for dual link, dual port is for redundancy, dual link is for increased bandwidth and it's just the link between the HBA and the expander, then any devices can use the extra bandwidth, including SATA.

Sounds like I have more reading to do on the subject!

 

[Chris Moore]

It is difficult to find good documents telling about this.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[judokan9]

Chris Moore yes this is true,

so I did a bit of testing.
I figured out that the IOM6 modules are dual link compatible. But you need to plug booth cables into the same IOM6 module, otherwise you only get 4 Lanes.

Show : camcontrol smpphylist /dev/ses0

37 PHYs:
PHY Attached SAS Address
0 0x500605b009224660
1 0x500605b009224660
2 0x500605b009224660
3 0x500605b009224660
4 0x500605b009224660
5 0x500605b009224660
6 0x500605b009224660
7 0x500605b009224660
8 0x5000cca02ad1a48d <NETAPP X412_HVIPC560A15 NA03> (da9,pass11)
9 0x5000cca02ad948f1 <NETAPP X412_HVIPC560A15 NA03> (da14,pass16)
10 0x5000cca02ad964a1 <NETAPP X412_HVIPC560A15 NA03> (da12,pass14)
11 0x5000cca02ad9b775 <NETAPP X412_HVIPC560A15 NA03> (da16,pass18)
12 0x5000c5006800752d <NETAPP X412_S15K7560A15 NA06> (da2,pass4)
13 0x5000cca02ad192cd <NETAPP X412_HVIPC560A15 NA03> (da7,pass9)
14 0x5000c5007156f1a5 <NETAPP X412_S15K7560A15 NA08> (da4,pass6)
15 0x5000cca02ad193d5 <NETAPP X412_HVIPC560A15 NA03> (da17,pass19)
16 0x5000cca02ad1ac79 <NETAPP X412_HVIPC560A15 NA03> (da5,pass7)
17 0x5000cca02ad93e99 <NETAPP X412_HVIPC560A15 NA03> (da15,pass17)
18 0x5000cca02ad1a325 <NETAPP X412_HVIPC560A15 NA03> (da13,pass15)
19 0x5000cca02ad193fd <NETAPP X412_HVIPC560A15 NA03> (da23,pass25)
20 0x5000cca02ad1aca9 <NETAPP X412_HVIPC560A15 NA03> (da8,pass10)
21 0x5000cca02ad1ecb5 <NETAPP X412_HVIPC560A15 NA03> (da18,pass20)
22 0x5000cca02ad1ab8d <NETAPP X412_HVIPC560A15 NA03> (da11,pass13)
23 0x5000cca02ad1ae31 <NETAPP X412_HVIPC560A15 NA03> (da21,pass23)
24 0x5000c5007e101025 <NETAPP X412_S15K7560A15 NA06> (da3,pass5)
25 0x5000cca02ad1a3cd <NETAPP X412_HVIPC560A15 NA03> (da19,pass21)
26 0x5000cca02ad19b6d <NETAPP X412_HVIPC560A15 NA03> (da25,pass27)
27 0x5000cca02ad1adc1 <NETAPP X412_HVIPC560A15 NA03> (da20,pass22)
28 0x5000cca02ad1eaa5 <NETAPP X412_HVIPC560A15 NA03> (da6,pass8)
29 0x5000cca02acfc865 <NETAPP X412_HVIPC560A15 NA03> (da10,pass12)
30 0x5000cca02ad3c735 <NETAPP X412_HVIPC560A15 NA03> (da22,pass24)
31 0x5000cca02ad195b9 <NETAPP X412_HVIPC560A15 NA03> (da24,pass26)
32 0x0000000000000000
33 0x0000000000000000
34 0x0000000000000000
35 0x0000000000000000
36 0x500a098001b440fe

0-7 are the 8 lanes to one IOM6 module.

Not surprisingly, there was no performance improvement here for me.

I have a pair of HB-SBB2-E601-COMP laying around too. They are also compatible with the DS4243, so I tested them for dual link capability and the results went out very well.
One surprising thing was, that they seeming to be a bit faster than the IOM6. I got around 10-15% faster read and write speeds with them.

 

[Chris Moore]

Not surprisingly, there was no performance improvement here for me.

I have two 24 bay backplanes in my server and I run one 4-lane SAS cable to each. It works well. Generally hard drives are not fast enough to saturate a 4-lane SAS cable because not every drive is talking at the same exact time and the controller (SAS expander) acts kind of like a traffic cop ensuring each device gets a turn. You could overload the bandwidth with SSDs and if you put enough spinning disks on there, but it takes way more spinners because they are comparatively slow. About 12 (give or take) SSDs will be all you can do without loosing some performance to a bottleneck but about four times (or more) that many spinning disks before you might be throttled by the SAS link.
Ballpark estimates from memory. Don't beat me up if they are not perfectly accurate.

 

[judokan9]

I have two 24 bay backplanes in my server and I run one 4-lane SAS cable to each. It works well.....

Exactly this is why i written it's not a surprise that there is no performance improvement.

Thank you all for your help and explanations.

 

[xnaron]

judokan9 - You have a similar setup to me (DS4246/IOM6/HB-SBB2-E601-COMP,LSI etc). In this thread https://forums.freenas.org/index.ph...246-ssd-speeds-and-ses-transfer-speeds.71234/ (TLDR) near the end I'm documenting an issue where my lsi cards 9200-8e and 9207-8e have an issue negotiating all the 6g eligible drives to 600MB/s. This isn't an LSI issue but most likely something with the DS4246/IOM6/HB-SBB2-E601-COMP. I realize you are running sas but you are connected with an lsi card. Have you ever tried sata drives? All of your sas drives negotiating at the highest speed?

I also have the HB-SBB2-E601-COMP controllers. I wish I would have gone with a 9300-8e at the start. It would have been better to invest in a 12gb controller for the future instead of spending on a few lsi cards and expensive QSFP to SFF-8088 cables. I haven't looked but I'm assuming the QSFP to SFF-8644 cables are similarly expensive. Ah well. :)