New ~72 TB FreeNAS build

[mikemccand]

Researching how to make Supermicro cases quiet I came across this interesting post: https://www.reddit.com/r/DataHoarde..._anyone_here_modded_a_4u_super_micro_case_to/

Which in turn referred to this interesting and scary post about using Norco cases:
http://wsyntax.com/cs/killer-norco-case

I had been considering Norco cases -- they are so much cheaper new -- and likely Norco has resolved the issues above, but it's quite spooky so I think I will try to get Supermicro case and make it quiet.

 

[mikemccand]

So I now have 10 12 TB Seagate EXOS drives and nothing to plug them into yet! I may smoke test them one by one with a spare Linux box ...

Apparently it is challenging to make Supermicro cases quiet, but it is possible, and I'd like to try. My current plan is to start from CSE-846BE1C-R1K28B which has silent power supplies already, and then swap out the case and internal fans to make it even quieter. I can't seem to find this case used though ...

I could maybe buy an older 846 case and swap out the power supplies for less $$$, but I don't understand all the tradeoffs with the older cases -- they seem to have different backplanes at least, and maybe slower SAS implementation. So I'll most likely just buy this case new.

I would really like to drive the fan speed by both drive and CPU temperatures, but from researching this doesn't sound so simple because the hardware normally controls the fan speed itself and it doesn't read the drive temps?

 

[mikemccand]

I spent some time learning about the complexities of SAS and backplanes (https://www.ixsystems.com/community...-sas-sy-a-primer-on-basic-sas-and-sata.26145/ was awesome, thanks @jgreco!) and I think I've got a good case and other components for my build:

CASE: Supermicro CSE-846BE16-R920B
- 24 drive bays
- BPN-SAS2-846EL1 backplane with single SAS2 expander
- quiet power supplies (2X PWS-920P-SQ), plenty of wattage (920)
- I think I only need one SFF-8087 to SFF-8087 cable to connect HBA to backplane?
- since I'm only populating 10 bays for now, I think I can swap in quieter internal and case fans to make the whole thing quiet enough

MOTHERBOARD: Supermicro X11SCH-F

RAILS: https://www.newegg.com/Product/Product.aspx?Item=N82E16816101354
https://www.wiredzone.com/supermicr...chassis-accessories-mcp-290-00060-0n-10021089 (my rack has rounded unthreaded holes)

CPU: Xeon E-2146G

RAM: 4X 16 GB ECC recommended by Supermicro: https://store.supermicro.com/16gb-ddr4-mem-dr416l-cv02-eu26.html

HBA: HP H220 https://www.ebay.com/itm/HP-H220-6G...-IT-Mode-for-ZFS-FreeNAS-unRAID-/162862201664

2X M.2 SSDs for boot/system drive: Crucial MX500 250GB https://www.amazon.com/dp/B077SL4FZG/ref=sspa_dk_detail_7?psc=1

2X mirror'd SSDs for mysql and maybe other fast zfs datasets: 2X 2TB Samsung 860 EVO: https://www.amazon.com/Samsung-500G...+ssd&qid=1557508501&s=electronics&sr=1-3&th=1
- I'll wire these to the motherboard's builtin SATA ports

Others:
- compatible CPU cooler, thermal paste, sata cables, mini-sas cable

Some questions:

* Will the latest FreeNAS work OK with this case (CSE-846BE16-R920B) and backplane (BPN-SAS2-846EL1)? Is that backplane too new? Will it handle my 12 TB Seagate Exos drives?

* The "Motherboards" listed for my case (https://www.supermicro.com/products/chassis/4U/846/SC846BE16-R920B) do not include any X11 boards, but does state that Mini-ATX is supported ... is this a problem?

* I have only SATA drives, which I think means there is no point in plugging in both Mini-SAS connectors from the HBA to the backplane?

* Can FreeNAS install and boot from mirror'd M.2 SSDs?

Thanks!

 

[Ericloewe]

Will the latest FreeNAS work OK with this case (CSE-846BE16-R920B)

It's a chassis, FreeNAS doesn't care.

backplane (BPN-SAS2-846EL1)?

It's LSI SAS2, perfect for FreeNAS.

Is that backplane too new?

Too new for what? The chassis?

Will it handle my 12 TB Seagate Exos drives?

It should, barring any freak incompatibilities (happens occasionally).

* The "Motherboards" listed for my case (https://www.supermicro.com/products/chassis/4U/846/SC846BE16-R920B) do not include any X11 boards, but does state that Mini-ATX is supported ... is this a problem?

Nope, not a problem with active CPU coolers. It could be a problem with passive coolers, but why would you go with that?

* I have only SATA drives, which I think means there is no point in plugging in both Mini-SAS connectors from the HBA to the backplane?

The backplane doesn't do dual-porting since it only has one expander (you'd need the dual-expander -EL2 backplane, second SAS controller and SAS drives for that to work). In your case, connect the second port if you want more bandwidth (not that you'd really need it without SSDs in the mix).

* Can FreeNAS install and boot from mirror'd M.2 SSDs?

Boot and install to yes. Install from, conceivably yes, but that seems convoluted and weird.

 

[jgreco]

The backplane doesn't do dual-porting since it only has one expander (you'd need the dual-expander -EL2 backplane, second SAS controller and SAS drives for that to work). In your case, connect the second port if you want more bandwidth (not that you'd really need it without SSDs in the mix).

Yeah, but he's talking about x8 wideporting, @Ericloewe ... x8 wideporting should work but I've had it act goofy in the past so I usually don't bother.

 

[Ericloewe]

Yeah, I forgot to mention that it doesn't always work, even though it should. Which is pretty weird, four lanes work fine but not eight? You've gotta wonder who messed that up and how.
Then again, it is the company that said that 32-bit LBAs would be enough for everyone sometime in the mid/late 2000s...

 

[mikemccand]

Thank you @jgreco and @Ericloewe -- sorry, I meant "install to" and "boot from", which sounds fine with mirrored M.2 drives. And I was worried the backplane might be too new for FreeNAS, but it sounds fine.

I think the one SAS expander is enough bandwidth for 10X spinning magnet disks.

I'm planning to put the SSDs on M.2 and SATA ports on the motherboard.

 

[LTCM]

To touch on the 4 4k plex issue. Freenas does not allow for hardware gpu transcoding. Really it means freebsd doesn't.

Other OSs do. If you do want to go back and re-attempt 4 4k transcodes just build another server for that purpose and stick a gpu in it. No need for ecc or enterprise grade equipment. Even a used i7 with quicksync would be a huge step up.

 

[mikemccand]

Thanks @LTCM -- a dedicated box for GPU transcoding sounds great -- a project for another day.

I've ordered all the parts and now the hardest part arrives: waiting!

 

[jgreco]

Nope, not a problem with active CPU coolers. It could be a problem with passive coolers, but why would you go with that?

Because the chassis is designed for passive coolers, and when you play things the way they were designed, you get fully-redundant push-pull fans cooling your CPU's; two in the front and two in the back. The failure of a single fan is a nonevent - there are actually FOUR fans cooling your CPU's.

In order for that to work you do need the shroud to be installed, so it is basically a matter of seeing if the shroud will work correctly.

Putting active CPU coolers in is a bad idea as it turns into a CPU bake when the single nonredundant fan on your cooler fails. Because you won't be able to install the cooling shroud with an active cooler, that doesn't save you either.

 

[mikemccand]

Because the chassis is designed for passive coolers,

Ahhh -- I missed that about the case. Thank you for clarifying. Now I have a challenge since I've got an X11 CPU coming ... hrmph. I guess I could live with the loss of redundancy and monitor the CPU temperature (and drive temps).