The best PCI 3.0 RAID controller for SSD

[poldas]

Hi

Which PCI 3.0 RAID cotroller do you recomend for 8x SSD storage?

 

[jon_2112]

Well you don't want a RAID controller. You need a controller that will simply present the disks to FreeNAS.

The 9211-8i is probably what you want. It has 2 SAS connectors, to which you can attach 4xSATA drives to each, via an SFF_8087 breakout cable.

 

[jgreco]

No, the 9211-8i will swamp out if you really want full SSD performance. It has a sad little PPC440 CPU which is completely adequate for HDD.

If you can put a bunch of the SSD's on the host motherboard SATA ports, that will relieve the pressure and a 9211-8i will perform reasonably well with two or four SSD's.

Otherwise the 9340-8i is probably a better choice.

 

[poldas]

I aleready have DELL R720xd and flashed H310 min to 9211-8i.

9340-8i will be much faster?

 

[jgreco]

The 9211-8i is powered by the LSI SAS2008.
The 9217-8i is powered by the LSI SAS2308.

SAS2008 has a PowerPC core at 533MHz.
SAS2308 has a PowerPC core at 800MHz.
SAS3008 has a PowerPC core at 1.2GHz.

Bear in mind that card performance probably does not scale linearly with clock.

 

[Stilez]

Edited slightly to reflect info on 2308 chip + STP by jgreco below - thanks!

Any PCIe 3.0 8x card, really, is the answer. (But HBA not RAID). Get the cheapest you trust. A full list of available cards and their controllers/specs is here at servethehome.com. (But dont skimp on the cables!!). Here's why:

ZFS likes HBAs rather than RAID controllers, so that it can directly manage the individual disks. But almost overwhelmingly, HBAs seem to be powered by the LSI/Avago/Broadcom 2xxx/3xxx chips, with 2008/2xxx (other than 2308) being PCIe 2.0 based with slower CPUs, and 3008/3108/3xxx (and 2308) being PCIe 3.0 based with faster CPUs. So those are really your choices simply because there aren't many others.

However because you only want HBA capability, we can say that all 2xxx based cards (except the 2308) are alike, and all 3xxx based cards are alike. (2308 based cards are closer to 3xxx when dealing with SATA SSDs, so I'll group 3xxx/2308 cards together for this question). Any other differentiating features wont matter, because yiu simply wont ever be using them in a ZFS based build. So the limiting factor on performance, and the *only* real differentiating factors, will be the number of devices supported, and the limits of the PCIe bandwidth itself.

These 8-device cards almost all have PCIe 8x connectors. A PCIE 2.0 8x link can handle 4.0 GB/s in each direction, and a PCIe 3.0 8x can handle about 7.88 GB/s bandwidth in each direction. For 8 devices that means:

• Almost all 2xxx cards except 2308 ( PCIe 2.0 8x) hosting 8 SSDs provide simultaneous peak PCIe bandwidth of about 500 MB/s per device each way.
• Almost all 3xxx + 2308 cards (PCIe 3.0 8x) hosting 8 SSDs provide simultaneous peak PCIe bandwidth of about 985 MB/s per device each way.

You dont say which SSDs youll use, but we can assume they are SATA 3 based (6 Gb/s), because you didn't highlight NVMe and because SAS SSDs just arent so common. SATA 3 maximum bandwidth is 600 MB/s and modern SATA SSDs can max out a SATA 3 link. (Because the 3xxx + 2308 cards use SAS 3, which allows for tunnelling of SATA plus higher 12 Gb/s SAS speeds, its possible that an SSD could have even higher than 600 MB/s bandwidth if being used by a SAS card, I guess -- I can't think how else to explain the 826 MB/s figure in one of the above posts!)

So the answer is simple. 8x modern SATA SSDs will have a good chance to swamp any 2xxx card/PCIe 2.0 8x connection (500 MB/s x 8 devices max) apart from 2308 based, although that would require all disks to be reading or writing simultaneously at peak which won't actually be so common due to timing and because many IOs will be smaller block sizes and limited that way. But it's quite possible. But they will have no chance at all of swamping any 3xxx or 2308 based card/PCIe 3.0 8x connection (985 MB/s x 8 devices max).

And so we can immediately say that all 3xxx and 2308 based cards with a PCIe 3.0 8x connection, will be effectively identical. So it really doesn't matter which of them you choose.

Get the cheapest. Like I said, a full list of available cards and their controllers/specs is here at servethehome.com. You can always crossflash to LSI firmware (the only ones I wouldn't are Supermicro's 3xxx based HBA cards, because they have that good a reputation generally for kit). You can also crossflash the RAID "IR" versions of these cards to HBA "IT" firmware as well.

But....

Get good quality cables! Do not skimp on those. Good cables will cost a lot. But I've had more issues with these cards due to poor quality 4 way fanout cables than anything else, and so have other people. Your best bet are cables by MicroSemi (used to be Adaptec, or took them over). Personally I pretty much would not buy any other brand. Their part number is 2280100-R ("ACK-I-HDmSAS-4SAS-SB-.8M"). Ignore the dangly ribbon cable attached to it!

You may also need SATA to MOLEX (old 4 pin) power connector adapters/splitters, because modern home user PSUs provide SATA connectors but these cables will require a MOLEX connector. Check that as well.

 

[jgreco]

HBAs seem to be powered by the LSI/Avago/Broadcom 2xxx/3xxx chips, with 2008/2308/2xxx being PCIe 2.0 based and 3008/3108/3xxx being PCIe 3.0 based.

Incorrect.

So those are really your choices simply because there aren't many others.

However because you only want HBA capability, we can say that all 2xxx based cards are alike,

Also incorrect.

The LSI 2308 is PCIe 3.0 and 50% faster.

Because the 3xxx cards use SAS 3, which is a superset of SATA and allows for higher 12 Gb/s SAS speeds, its possible that an SSD could have even higher than 600 MB/s bandwidth if being used by a SAS card, I guess

ALSO incorrect.

SAS is Serial Attached SCSI and is something completely different than SATA, which is a descendant of IDE and ATA. SATA devices work on SAS ports directly because the controller switches to speaking SATA instead of SAS, and through expanders thanks to something called STP, Serial ATA Tunneling Protocol, which allows SATA to be tunneled over SAS.

There is no turbo boost that will allow SATA devices to work at faster than 6Gbps even if you move on up to SAS3. The one thing that works better is if you have a SAS3 HBA and expander, you can have more bandwidth to your drive shelf -- you get 12Gbps * 4 between the HBA and the expander.

The main problem with using SSD behind an HBA is that you do have the potential to swamp out the CPU on the HBA's; see above for speeds. LSI released the 2308 in large part as a response to the capping effect on the 2008 when you either have a huge number of HDD's or fast SSD's. The 2308 is generally expected to be "as fast as 6Gbps can go" though if you showed me a graph showing a 3008 as a handful of percent faster, I wouldn't be shocked.

 

[Stilez]

Thanks for this, jgreco. I wasn't aware of the speed nudge on the 2308, and its good to be corrcted. But overall I don't think your info changes my answer. The OP asks about PCIe 3.0 cards, and your answer simply means that 2308 based cards are grouped with those and not, as I had understood, PCIe 2.0 cards.

The rest really wouldn't change anything.

• The PCIe 3.0 based cards are still all based on processors + PCIe speed likely not to be swamped by 8 x fast SSDs, and the PCIe 2.0 cards are still based on processors + PCIe speed that are likely to be capable of being swamped. The only change is that 2308 is now a PCIe 3.0 card not a PCIe 2.0 card - thanks, that's worth knowing.
• The details of how SATA can work on a SAS controller are interesting but don't change the outcome - in effect it works "as if" SATA is a subset of SAS, the fact theres in fact a tunnelling protocol is informative and a worthwhile correction but the outcomes are unchanged..
• The CPU swamping is pretty much going to be taken care of automatically - all PCIe 3.0 cards seem to have sufficient CPUs, so unless you switch back to a PCIe 2.0 based card (which the OP explicitly excludes), swamping the CPU shouldn't ever be an issue with any ofg the PCIe 3.0 cards or their controllers: none of the PCIe 3.0 cards have CPUs able to be swamped by 8x fast SATA.

I do appreciate the finer points of correction. It's useful to learn. But in terms of an answer, with the sole exception that one of the 2xxx cards is not PCIe 2.0 and swampable CPU, but should properly be classified with the 3xxx cards as PCIe 3.0 and faster CPU, the rest still stands. I've edited my answer to ensure the minor points of information are accurately presented though, for avoidance of doubt.

Thanks again - good detail!

 

[jgreco]

The only change is that 2308 is now a PCIe 3.0 card not a PCIe 2.0 card - thanks, that's worth knowing.

AND the 2308 has a 50% faster CPU. Given the sorts of complaints I've seen about the 2008 topping out around 500MBytes/sec per device, and that the 2308 was specifically released over the PCIe 3 and SSD concerns, that's relevant.

On the other hand, since even a relatively fast FreeNAS host with a 40Gbps uplink and 8 SSD's still only works out to about 5Gbps per drive, you're really not that likely to notice any of this. Scrub or rebuild speeds are when this MIGHT be relevant. Local operations (jails, maintenance, whatever) on the host will feel wicked fast regardless of the HBA, even if it is getting swamped. And it's worth noting that most SSD's do not really deliver "max" performance in real world conditions, so actual swampage may not be a meaningful concern.

I would look to the 3008 simply as a way to future-proof if you are intending to do a large shelf of drives on an expander. Once you get out past 8 devices, yes, there's a compelling argument for the 12Gbps.

The odd variable here is the OP's specification of PCIe 3.0. I guess my original answer was wrong because the 9211-8i was never a relevant choice (PCIe 2.0), but rather a reaction to the previous comment. If you absolutely require PCIe 3.0 and will never have more than 8 drives, LSI 9217-8i (SAS2308) is the way to go. If you want to be realistic and will never have more than 8 drives, drop the PCIe 3.0 requirement and go to PCIe 2.0 on the LSI 9211-8i and save some money. If you don't care about money, then the SAS3008 is the way to go. It's fast and more future-proof, but about 10x the cost.

 

[poldas]

Otherwise the 9340-8i is probably a better choice.

I have DELL R720xd.
1. Can I get cable which is fitting to 9340-8i and my chassis
2. Can I buy IBM M1215 and flash it?

 

[jgreco]

Having not seen an R720xd, I simply don't know. If it uses standard cabling, probably.