Hardware Question with 3+ NVMe M.2 SSD ZFS (PCIe 3.0/4.0)

[pixelwave]

I want to build a TrueNAS with at least 3 (ideally 4) NVMe M.2 SSDs with full PCIe 3.0 or better 4.0 support / speed.

The options I read so far are to go with an ASUS Hyper Card M.2 and compatible CPU+Mobo that support bifurcation on a PCIex16 slot to switch to x4x4x4x4 operation mode. To find that combo ... it is quite difficult because most do not advertise this and sometimes it seems a hit and miss.

The alternative would be an active PCIe x16 card with 4x M.2 slots (like Sonnet, IOCreast) are there any experiences with TrueNAS Scale? Are those compatible to use for ZFS?

 

[HoneyBadger]

Most modern server motherboards will advertise their support for PCIe bifurcation in their manual - if you've got a particular board or platform in mind, reading the BIOS/EFI section of the manual will usually shed light on it. Certain OE servers will also be well-known as supporting it.

"Active" PCIe splitter cards have a PCIe switch chip installed that behaves in a manner similar to a SAS expander or network switch; they do work with ZFS, but the drawback is that this chip adds to the cost of the device. Some cards also oversubscribe the downstream lanes - it may advertise support for 4 x4 M.2 slots, but only have an x8 connection to the motherboard. Check the spec sheet carefully. In addition, I'm not aware of a PLX chip that does PCIe 4.0 yet, so you'd be limited to PCIe 3.0 - although you're likely to hit a bottleneck elsewhere in the system first.

 

[NickF]

This is an interesting topic that I have been exploring quite in depth, although I dont have final numbers to share just yet.

Some interesting things of note can be found in my testing which I have posted here https://forum.level1techs.com/t/truenas-scale-performance-testing/187486

I intend do some more thorough testing with my new H12SSL-I and Epyc 7282 and bifurcation vs using a card with a PLX chip. Initial impressions are that it seems that the PLX chip-based cards are at least as fast as the bifurcation being done on the motherboard side.

The TLDR so far is that at least in terms of FIO testing, you should set your expectations appropriately and its probably not going to be as fast as you would like.

Honeybadger is also correct, I also have not seen any PCIE 4 based PLX switch chip cards.

 

[fenichelar]

I am currently using the ASUS Hyper AOC with 2 mirrored PCIe 4.0 SSDs (Seagate Firecuda 530) on my Supermicro X12DPi-N6. Works great. I had to change the the PCIe slot from Auto to 4x4 in the BIOS. After performing a BIOS update, it reverted the setting which caused TrueNAS to think the drive was unplugged. After applying the BIOS setting again and a quick resilver, the pool was back online.

I have a 3rd Seagate Firecuda 530 on the motherboard. The one on the motherboard has the Seagate heatsink. I have a 45Drives Q30 chassis. The SSD on the motherboard is 30 degrees at idle. The two in the AOC are 21 degrees, which is ambient. I have been very pleased with the cooling performance of the ASUS Hyper AOC.