Comments / advice on HBAs and SSDs

[MountainMan]

Hello...

I’ve read the Hardwire Recommendations Guide, but am hoping for some input on a couple specific pieces.

Basically I’m trying spec and gather parts for a TrueNAS Scale based server to replace a couple aging hardware RAID systems at home. They are used for NAS storage (Plex, family videos, photos), family docs, desktop backups, and my development work. They also run about 20-25 containers for services/applications (Plex, email, DBs, Home Assistant, Grafana, NGINX, etc) and a few VMs.

It will be a fresh start, but I have picked up a few items as nice deals have come up:

• Chenbro RM31616M3-R875 - 3U 16 bay chassis w/ 12G backplane (4 x SFF-8643) and redundant 875W PS
• 5 x Red Pro 14TB drives
• CyberPower 1500VA rack UPS

I will also have one piece from my current system I was going to re-purpose:

• Sans Digital 8 bay external cabinet (2 x SFF-8088) with 8 x 3TB WD Red drives — which I was just going to use as my "backup" pool.

In the end I’m looking to have 3 pools:

• Storage pool - 2 x mirrored pairs of WD Red Pro 14TB + hot spare
• App/VM pool - 1 x mirrored pair of 2TB SSDs + hot spare
• Backup pool - re-purposed external 8 x 3TB WD Red using RAIDZ1

I’m currently contemplating a 12 series SuperMicro MB for (future) PCI 4 use — to get a little more life out of the base build. Although CPU options may lead me back to an 11 series board. I’m thinking I'll need 96G RAM (~50G for TrueNAS/ZFS + 32G+ for apps/VMs) but perhaps 128GB would a better configuration performance wise(?). TBH server RAM configuration options beyond ECC (RDIMM, LRDIMM, ranking, socket use, etc) have my head spinning — but that’s probably a different post once I settle on the MB — I just know I’ll need a bunch :)

Here is what I’m trying to figure out at the moment though:

1. HBA (internal 12G - 16 port) - Since the chassis has a 12G backplane I figured I’d go with a 12G adaptor in case I pick up some SAS drives down the road. With LSI I was looking at the 9400-16i (PCI3) or potentially 9500-16i (PCI4), but they both seem incredibly hard to source from normal vendors. Only small dealers and eBay sellers in HK and China? After further looking I came across the Lenovo ThinkSystem 430-16i and 440-16i, which as far as I can tell are re-branded 9400-16i and 9500-16i ? The 430-16i can be had for ~$420 right now, so I was leaning toward that, but searches for verified TrueNAS compatibility seem very limited. I feel like it’s a safe choice, but is it? Would flashing to "IT" firmware be required even though it’s a non-raid HBA?

2. HBA (external 6G - 8 port) - This would just be for the external backup pool. I was looking at a 10Gtek SAS 9200-8E which is basically a LSI SAS2008/9200-8E compatible card at a good price ($145), which according to reviews, is fine with TrueNAS. Unless there is some reason to avoid or a better option?

3. SSDs for App/VM pool - While I want to use SSDs to keep this pool snappy for app/services/VMs, the data will all be backed up and in the end isn’t as important as the family movies/photos regardless. So I wasn’t planning on breaking the bank for SLC SSDs, figuring I could use more reasonable options like the WD Red SA500 2TB or Samsung EVO 870 2TB and toss a hot spare at it for good measure. With 1200-1300 TBW ratings, ashift=13 and SMART stat monitoring I feel relatively safe, unless someone can convince me I’m just fooling myself ? :) With that in mind, I’ve always used Samsung in the past on many systems, and never had a failure after quite a bit of write abuse, but am open to other options. The WD Reds just caught my eye with a little higher TBW and I’ve had good experience with their spinning disks. Any thoughts or other good options?

4. System drive(s) - I was planning on a separate mirrored 500G SSD pair for the TrueNAS system itself. While I should have 8 open bays, I’d rather toss the pair inside the chassis and use a couple SATA ports off the MB, keeping my bays open for future pool expansion. Is using the SATA ports off something like an Intel C252 or C236 controller possible, not possible, or just a ridiculously bad idea ?

I was hoping this wouldn’t get long, but it did, sorry about that. Anyway, thanks in advance for ANY comments, suggestions, or straight up criticism. ;)

-James

 

[NugentS]

1&2 I shall let other comment as I still haven't got my head around the somewhat bewildering model numbers of LSI cards. I thought the 94xx came under the heading of not good - but others will know more.

3. I used Crucial MX500's for a while. I reckon they would have lasted 4-5 years for half a dozen or so VM's before wearing out, which I was fine with. Do not use Samsung QVO's which are cheap for a reason

4. 500GB is way bigger than needed. Just get a couple of small 32+GB drives off Ebay and use those.

 

[MountainMan]

1&2 I shall let other comment as I still haven't got my head around the somewhat bewildering model numbers of LSI cards. I thought the 94xx came under the heading of not good - but others will know more.

3. I used Crucial MX500's for a while. I reckon they would have lasted 4-5 years for half a dozen or so VM's before wearing out, which I was fine with. Do not use Samsung QVO's which are cheap for a reason

4. 500GB is way bigger than needed. Just get a couple of small 32+GB drives off Ebay and use those.

Thanks for the comments... :)

Agree on the QVO. Actually thinking about the app/VM pool more, it seems like I overlooked the option to use a pair of 2TB NVMe via PCIe adaptor. For roughly the same cost I'd get the same endurance but much better performance.

 

[Etorix]

I’m currently contemplating a 12 series SuperMicro MB for (future) PCI 4 use — to get a little more life out of the base build. Although CPU options may lead me back to an 11 series board. I’m thinking I'll need 96G RAM (~50G for TrueNAS/ZFS + 32G+ for apps/VMs) but perhaps 128GB would a better configuration performance wise(?). TBH server RAM configuration options beyond ECC (RDIMM, LRDIMM, ranking, socket use, etc) have my head spinning — but that’s probably a different post once I settle on the MB — I just know I’ll need a bunch :)

Simple:
Xeon E3, E-2000, W-1000 = "Core + ECC" = UDIMM
Xeon D, E5, W-2000, W-3000, Scalable = RDIMM
most RDIMM plateforms also support LRDIMM, which has even higher capacity, but you cannot mix: it's either all RDIMM or all LRDIMM
(L)RDIMM is the way to go for high RAM, as second-hand modules are more affordable than UDIMM

Here is what I’m trying to figure out at the moment though:

1. HBA (internal 12G - 16 port)

Is there a SAS expander or do you need to plug the four connectors?
The 9400 is not as thoroughly tested than older controllers. 9300 is reportedly the well-supported way to go.

4. System drive(s) - I was planning on a separate mirrored 500G SSD pair for the TrueNAS system itself. While I should have 8 open bays, I’d rather toss the pair inside the chassis and use a couple SATA ports off the MB, keeping my bays open for future pool expansion. Is using the SATA ports off something like an Intel C252 or C236 controller possible, not possible, or just a ridiculously bad idea ?

SATA ports from the chipset are perfectly fine.

 

[mrpasc]

Thanks for the comments... :)

Agree on the QVO. Actually thinking about the app/VM pool more, it seems like I overlooked the option to use a pair of 2TB NVMe via PCIe adaptor. For roughly the same cost I'd get the same endurance but much better performance.

Make sure the motherboard of your choice supports bi-furcation for PCIe slot, otherwise you must go for PCIe adaptor with built-in PCIe switch which would make a significant surplus to the costs.

 

[MountainMan]

Simple:
Xeon E3, E-2000, W-1000 = "Core + ECC" = UDIMM
Xeon D, E5, W-2000, W-3000, Scalable = RDIMM
most RDIMM plateforms also support LRDIMM, which has even higher capacity, but you cannot mix: it's either all RDIMM or all LRDIMM
(L)RDIMM is the way to go for high RAM, as second-hand modules are more affordable than UDIMM

Excellent, thanks. Are there any sources for second-hand (L)RDIMMs you might recommend?

Is there a SAS expander or do you need to plug the four connectors?
The 9400 is not as thoroughly tested than older controllers. 9300 is reportedly the well-supported way to go.

No expander I'm afraid, that's in the "E" version rather than the "M" I have. Although it's also almost 3x the $ I picked up the M for, so I don't feel too bad. :) So I would be using all 4 connectors. I haven't really searched on the 9300's yet, but will take a look.

SATA ports from the chipset are perfectly fine.

That helps. Between that, and changing the my original SSD plan to an NVMe pair, I should have a lot of open bays which is great.

 

[MountainMan]

Make sure the motherboard of your choice supports bi-furcation for PCIe slot, otherwise you must go for PCIe adaptor with built-in PCIe switch which would make a significant surplus to the costs.

Yep. Thanks again. Definitely headed that direction.

 

[Etorix]

Excellent, thanks. Are there any sources for second-hand (L)RDIMMs you might recommend?

The forum at ServeTheHome.com, eBay and whatever small ads site is relevant for your local market.