SuperMicro HBAs to BackPlane questions

[nephri]

Hi all,

I'm currently under discussion with a seller for buying 2 chassis made like this:
- Chassis : SC846E1-R900B
- PSU : 2x 900W PWS-920-1R
- BackPlane : BPN-SAS2-846EL1 (replace the original SAS1)

I would change the PSU with a SQ model, but it's not the purpose of this post.

I readed the BPN-SAS2-846EL1 documentation because i want to be sure that this backplane will not be the component responsible of any throughput limitation in the final build.

So, i'm not really familiar with backplane and expanders and i saw it has 3 primary connectors called PRI_J0, PRI_J1, PRI_J2 but i don't really found any documentation of how theses connectors works and can be used.

I would know if i can connect 3 HBA to the backplane ? Or connect a single HBA to the backplane using 2 or 3 connectors ?

I understand that i can't use failover because the EL1 don't have a second expander chip. But this not a mandatory feature and since i will probably use SATA drive (enterprise class), it's not an issue.

So, finally i don't found what is possible to do and what is not with HBA <---> BPN-SAS2-846EL1 .
I would have your welcome advices on which architecture i should do for have a 24x hard drives with the maximum throughput.

I have to say that i have plan to use a dual 10 Gb NIC using SFP+ connector. But if all goes well i will maybe able to use a single 40 Gb NIC using QSFP. (but it's largely unsure at this time).

- Which HBA i should use (only one or more than ones ) ?
- Which cables i should use (it seems to be the SFF-8087 from documentation i saw) ?
- How many cables ?

The SFF-8087 seems to be a 4x SAS channels at 6Gbs/s = 24 Gbs/s.
It would be sufficient for a dual 10Gb/s NIC but not for a single 40Gb/s NIC if i use only one cable.

I would also know if it has a real advantage to go with a SAS3 backplane (12 Gb/s) ?
With SATA drive at 7200 RPM, it doesn't seem to be an high plus-value, have missing something ?
With SSD it should be significantly different.

I do this post because i don't want make a mistake with this buy because it represent a significant cost i want to be sure where i go. Maybe i'm having wrong questions...

Thanks in advance for your helps.
Best Regards,
Sébastien.

PS: sorry for my poor english, i'm french.. (i know it's not an excuse :p)

 

[maglin]

You will have only one HBA Connected to that backplane. It's in the documentation. You can cascade more expanders on it.


Sent from my iPhone using Tapatalk

 

[jgreco]

You may want to verify with Supermicro that your revision of the SC846 will be upgradeable to the SAS2 backplane.

https://forums.freenas.org/index.ph...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

The three SAS connectors (PRI_J0...J2) can be used for upstream or downstream connections. They're just 4 lane wideports to the SAS expander. You can attach one of them to an HBA and leave two empty. You can attach one of them to an HBA and then extend two of them to back panel connectors to connect more shelves. You can attach one of them to an HBA and extend two of them to back panel connectors to connect to other servers, creating a shared storage topology. Etc.

 

[nephri]

Some guys on another forum says that we can do a 'dual link' between the HBA and this backplane for having 8 lanes instead of 4.
Have you any feedback with such feature ?

Best regards,
Sébastien.

 

[jgreco]

I did experiment with x8 wide-porting but was never able to get it to work correctly - the addition of the second set of lanes actually reduced performance. I chalked it up to maybe needing updated firmware on the expander or HBA, or possibly a configuration issue. Since it isn't likely that a 12-drive RAIDZ3 vdev will be exceeding 24Gbps on a NAS that only had 20Gbps of network connectivity, figuring this out wasn't of much interest here.

 

[nephri]

It's right, for a dual 10Gb NIC, 4 lanes (24 gb) is enough.
But if i'm plan to use a 40Gb NIC (QSFP), i could be an issue and "x8 wide-porting" would be a good alternative if it works out of the box.

In this case, maybe i would go on SAS3 backplane (that cost very much). In case of SAS3, the same cable SFF 8087 wire ate 12 Gbs instead of 6Gbs or it requires some other cables ?

Another question : I plan to buy a motherboard like X10SRH from supermicro. can i use the SAS3 LSI chip connected to the SAS2 backplane without issue (but negociated at 6Gb/s) ?
Or should i go on an external SAS2 HBA ?

Best regards,
Sébastien.

 

[jgreco]

It's right, for a dual 10Gb NIC, 4 lanes (24 gb) is enough.

That's oversimplifying it, but close enough.

But if i'm plan to use a 40Gb NIC (QSFP),

You're not likely to squeeze 20Gbps out of a FreeNAS box without significant engineering effort and expertise. I wouldn't worry about 40Gbps unless you're going all-SSD, in which case you're looking at something more like the SC216, not an SC846.

i could be an issue and "x8 wide-porting" would be a good alternative if it works out of the box.

Which it doesn't seem to, but, again, didn't do a deep-dive on the topic.

In this case, maybe i would go on SAS3 backplane (that cost very much). In case of SAS3, the same cable SFF 8087 wire ate 12 Gbs instead of 6Gbs or it requires some other cables ?

No, then you need a SFF-8643 to SFF-8087 cable. See the SAS primer. They're readily available and they work fine.

Another question : I plan to buy a motherboard like X10SRH from supermicro. can i use the SAS3 LSI chip connected to the SAS2 backplane without issue (but negociated at 6Gb/s) ?
Or should i go on an external SAS2 HBA ?

That mostly depends on the state of the LSI drivers in FreeNAS. Hardware-wise, it would work fine, but most people in FreeNAS-land are still using SAS2 HBA's and that's what you're more likely to be able to get help with if there's trouble.

 

[nephri]

Thanks, i readed the SAS primer but i skipped the parts of SFF-8643 !! oups

Ok, i will go mainly for X9SRH instead of X10SRH. the X9 comes with SAS2 chip and i will use SFF-8087.
I had a 40 Gb NIC (and i will test it maybe later) but still go with dual 10 Gb as long as i will not have many disks at the beggining.
When i will be more confident with FreeNas and FreeBsd, i will go further into.

Best regards,

 

[jgreco]

Good luck and let us know how it goes.

 

[nephri]

Regarding cost of this build, i would not rely only on luck :-/

So, for the moment i go on:
- Chelsio T3 dual 10gb/s NIC (i have a Mellanox 40Gb Nic but it not seems to be on FreeNas 9.3 compatibility list)
- SC846 chassis with a SAS2 backplane
- PWS-1K28P-SQ PSU
- X9SRH motherboard including the SAS2 chip
- Xeon E5-1620 v2
- 32 Gb of RAM (for start). I don't know with which topology. I don't checked if it's better to put 1 bank, 2 or 3. Any advices ?
- A Gnodal GS4008 switch for handling 10Gb/s network
- A LB4M switch (with a dual 10Gb/s) for handling Ethernet RJ45 gibabit network
- and few disks (i hadn't selected which for now...)

I bought 2 SC846 for a backup storage solution. But for now, it will be "empty", because i haven't the budget for it and my wife start to cry about my expense...

 

[depasseg]

I plan to buy a motherboard like X10SRH from supermicro. can i use the SAS3 LSI chip connected to the SAS2 backplane without issue (but negociated at 6Gb/s) ?

Every SAS3 (3008 based) controller I've seen connected to a SAS2 backplane (those with a built-in expander, meaning not the TQ type) has had issues of some sort. Even in my server, I ended up getting an external SAS2 HBA for the SAS2/SC846. The internal SAS3 controller to SAS3 backplane is fine.

 

[nephri]

Every SAS3 (3008 based) controller I've seen connected to a SAS2 backplane has had issues of some sort. Even in my server, I got an external SAS2 HBA for the SAS2 (SC846). The internal SAS3 controller to SAS3 backplane is fine.

Thanks for this feedback. I didn't find at this time cheap supermicro chassis including sas3 backplane. Even a backplane sell separately are difficult to find. So i will continue to go on SAS2 version and i will probably buy a m1015 or a X9SRH instead of the X10SRH.
i would love to try SAS3 instead SAS2 but it will be out of my budget...

 

[jgreco]

Every SAS3 (3008 based) controller I've seen connected to a SAS2 backplane has had issues of some sort. Even in my server, I got an external SAS2 HBA for the SAS2 (SC846). The internal SAS3 controller to SAS3 backplane is fine.

Really? That's strange. Haven't had any problems here with the 3108 (not for FreeNAS obviously). Admittedly I think we're only using it in one recipe; when looking around for a new hypervisor design I really wanted to go 2U. We already use the X10SRW board that's found in the 1018R-WC0R so I kinda did a modern UP version of the 2027R-WRF by integrating a SC213A-R740W with the X10SRW. Throw in a Supermicro S3108L-H8iR for 2GB cache goodness, add the supercap/TFM module for writeback goodness, and yay. Loaded up with an E5-1650v3 you can put together a really nice not-totally-wattburn hypervisor. We've been hitting the local datastores on it fairly heavily without any trouble, which is a SAS3 controller to SAS2 backplane scenario. It doesn't, however, have an SAS2 expander. Is there any chance that maybe what you're seeing is SAS3 controller vs SAS2 expander barf?

 

[depasseg]

Is there any chance that maybe what you're seeing is SAS3 controller vs SAS2 expander barf?

Yes, this is probably a better recap of the issue, I'll update my post. But the bottom line is that my on board 3008, didn't cooperate fully with the SC846 (missing drive - I think due to enclosure reporting getting mixed up with drive IDs), and some strange performance slowdown issues under heavy transfers.

 

[jgreco]

Wonder if that might be driver issues though.

 

[depasseg]

Absolutely think it's a driver issue, but unfortunately Supermicro is taking their sweet time providing the upgraded version.

 

[jgreco]

That makes more sense to me than having it be an underlying hardware issue.