New ~72 TB FreeNAS build

[mikemccand]

Hi All,

First post!

I've got a ~7 year old FreeBSD ZFS 4U rack-mount home filer, 5 4 TB drives in Raid Z1, 32 GB RAM, that I'm looking to replace with a new build.

It has served me well, despite non-ECC RAM, and despite only RAIDZ1, but is getting too close to full, and has been showing more data errors recently. I have backups to another ZFS box via "zfs send/receive".

It's holding media, data sets, a growing MySQL DB. It doesn't need to be super high performance. I don't think I need separate ZIL or L2ARC cache devices.

I'd like the new box to have 64 GB ECC RAM, hoping to get ~4 X the storage, maybe with 8 12 TB NAS/DC drives in RAIDZ2.

Tentative parts:

- Case: either one of the Supermicro 4U rack mount cases, or the cheaper non redundant power supplied Rosewill cases: https://www.newegg.com/Product/Product.aspx?item=N82E16811147164

- Motherboard: Supermicro X11SCH-LN4F (it's modernish, supports 8X SATA native, recommended in the FreeNAS hardware guide)

- Boot drives: 2X mirror'd boot SSDs: maybe Samsung EVO

- HBA: https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664

- 2X Mini-SAS to 4X Sata cables: https://www.amazon.com/CableCreation-SFF-8087-Female-Controller-Backplane/dp/B07CKX62L6

- Hard drives: 8X 12 TB WD Gold, or WD Ultrastar DC HC520 12 TB HDD

- CPU: ?

- RAM: 2X 32 GB ECC UDIMM

Questions:

- This motherboard has 2 M.2 connectors -- could I use those for my 2X mirrored SSD boot drives?

- If I wanted to use 10 (instead of 8) drives, I think I would need a SAS expander; does anyone know which SAS expander works well with the above card?

- How to know which HBA card to pick? The Art of Server sells so many on Ebay: https://www.ebay.com/str/theartofserver/HBA-IT-mode-controllers/_i.html?_storecat=23844485012

- Which CPU? I'm overwhelmed by all the Xeons out there ... I'd like something beefy enough to transcode maybe up to 4 4K video streams from Plex, but fast single core clock speed too.

- Which specific RAM? Too many choices... and the "Tested Memory List" on Supermicro's site only lists 8 and 16 GB UDIMMs. Hmm are there even 32 GB ECC UDIMMs?

- Is it too soon to trust the 12 TB NAS/DC drives?

- Is it better to put the drives all on the HBA? Or on the 8X native SATA ports?

Thank you for any help! This will be my first FreeNAS build.

Mike

 

[Constantin]

I'd consider

• a used case on ebay. Plenty of CSE 836 abound, at very reasonable prices. The power supplies are ludicrously cheap.
• using m.2 for SLOG, if the need ever arises. (P4801x)
• going embedded ...the X10SDV-2C-7TP4F is very inexpensive for what it offers. (SFP+, 16 SATA3/SAS2, two PCIe x8, One m.2 @ 3.0x4, etc.)

The only negative I've encountered with that series is that it is very finicky re: memory. Be sure to buy the exact right spec (including the RAM chip die revision generation) or you may have a no-POST like I did.

 

[Chris Moore]

Case: either one of the Supermicro 4U rack mount cases, or the cheaper non redundant power supplied Rosewill cases: https://www.newegg.com/Product/Product.aspx?item=N82E16811147164

I highly recommend repurposing enterprise server gear that is being sold to the public at a fraction of the new cost. I can provide some suggestions if used gear would be a option for you

 

[mikemccand]

Thank you for the suggestions -- repurposing used case/power supply/other parts is compelling. Do power supplies get "old" and more likely to fail over time, even in these redundant server cases? If that is not much of a risk, I think the used case is very compelling -- massively cheaper.

X10SDV-2C-7TP4F looks compelling too -- I never heard of "Flex ATX" size motherboard. But only 2 core / 4 threads CPUs / 25W TDP? Is that enough for 4 4K Plex real-time transcoding streams?

 

[Constantin]

That seems unlikely. I didn’t see that spec. I’d research the transcoding specs. IIRC, you need a pass mark of like 17,000 per stream. I’d like to think that a 4+ core CPU with a 17k passmark per core might work yet get expensive / hot.

 

[Chris Moore]

But only 2 core / 4 threads CPUs / 25W TDP? Is that enough for 4 4K Plex real-time transcoding streams?

I would not expect it to be enough.

Do power supplies get "old" and more likely to fail over time, even in these redundant server cases? If that is not much of a risk, I think the used case is very compelling

The components do tend to become less efficient over time, but many of the server systems I deal with were built with Platinum efficiency power supplies to begin with and they hold up very well over time. I have run servers (where I work) for over a decade with no problem at all. The components are usually very stable.

This one looks to be a good value:
https://www.ebay.com/itm/Supermicro...-24-x-HDD-Storage-Server-W-Rails/173688157339
It is currently configured with a hardware RAID card that would need to come out and be replaced by a SAS controller like this:

https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664

Other than that, you might decide you want faster processors, but that is an easy change to make later. This server can take any of the Xeon E5 V2 processors and I can make some suggestions there if you like.

 

[Chris Moore]

Is it too soon to trust the 12 TB NAS/DC drives?

Were I work, we just ordered a server with sixty of the Seagate Exos 12TB drives in it and 86 more drives of the same model to upgrade other servers we already have. We decided to go to the 12TB drives because of how well the 10TB drives we got last year are working out. I think they are ready.

This is what we just ordered: https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST12000NM0007/dp/B0759Q9FXZ

These are the ones we ordered last year: https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST10000NM0086/dp/B01LXXV880

PS. For my home NAS, I use a system like this:
https://www.ebay.com/itm/Chenbro-Cl...8-Bay-Enclosure-Chassis-16gb-RAM/382905905586
with the memory upgraded to 64GB and the H220 SAS controller. It works really well. A nice feature of the SAS Expander backplanes in this and in the rpevious model I suggested, they have a locate LED that you can turn on to find the drive you need to pull when there is a drive fault.

 

[Chris Moore]

Which CPU? I'm overwhelmed by all the Xeons out there ... I'd like something beefy enough to transcode maybe up to 4 4K video streams from Plex, but fast single core clock speed too.

A processor like this would work to upgrade either of the two systems I suggested.
https://www.ebay.com/itm/Intel-Xeon...-25MB-8GT-s-130W-Processor-SR19W/113679000767
In the dual socket system, you would need two matching processors.

 

[Constantin]

Careful, the passmark for that processor is "only" 10K.

If multiple 4K transcodings are really necessary, I'd research the processor requirements very carefully and then proceed. Any CPU that can handle ECC RAM and your transcodings will scoff at the load imposed by the FreeNAS. A good HBA in IT mode can be had from eBay for like $60 and then you have the choice of running SAS or SATA drives, etc.

I'm kind of in awe at the processor specs that PLEX suggests are necessary for 4K HD transcoding. These kinds of requirements suggest dedicating an entire processor per stream. Among the most cost effective CPUs that meet the spec are the Ryzen series (see Ryzen 7 2700X Pro, for example), and those cost (used on eBay) , $300 a pop. So not only only would you have to find a quad-Ryzen rig (if that even exists), but you'd also have to spend $1200 just on 4 used processors.

Another option is adding GPUs to do the hardware acceleration for you. That too will require some careful review. However, this may be the more cost effective solution - it may also allow you to use a lower power board like the embedded one I referenced in conjunction with a powerful GPU. That way, the FreeNAS can idle at low power most of the time and as needed the GPU will spool up to provide the necessary transcoding.

Good luck. I have yet to graduate beyond 1080P, which is at the limit of my visual acuity anyway. As a result, no transcoding for my rig.

 

[Jailer]

Another option is adding GPUs to do the hardware acceleration for you.

This isn't supported currently by FreeBSD, it's supposed to be added in 12.0.

Your best bet for dealing with 4K content is to make sure you have clients that support direct play of the content. Transcoding 4K takes way too much CPU power at the present time IMHO to consider multiple transcoded streams.

 

[Chris Moore]

Good luck. I have yet to graduate beyond 1080P, which is at the limit of my visual acuity anyway. As a result, no transcoding for my rig.

I use a single Intel Xeon E5-2650 V2, 2.6GHz 8 Core (16 thread) processor in my system for Plex and it does all that I need, but I have only run three streams at once...
The thing that I do is pre-transcode everything to 1080p. That way Plex can (my term) up-sample when it is feeding my 4k HDR TV and down-sample for anything that is smaller than 1080p. It almost never goes to 80% on the CPU and I have had zero problems with it.
I have done side by side tests and I can't see the difference between media saved as 4K and media saved as 1080p when I am sitting at a normal viewing distance.
Here is a link to an article about that:
https://www.nbcnews.com/technology/...phones-surpass-limits-human-vision-2d11691618

and another article if one needs more "evidence"...
https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stupid/

 

[Constantin]

Your best bet for dealing with 4K content is to make sure you have clients that support direct play of the content.

That's been my approach, albeit usually for upscaling 480/720P content to 1080P.

Transcoding 4K takes way too much CPU power at the present time IMHO to consider multiple transcoded streams.

That's the conclusion I am coming to also. The spec requirements are intimidating.

 

[mikemccand]

OK I will back off of the 4K streams transcoding requirement :) It sounds too ambitious. I'll just pre-encode to 1080p or 4K; that's essentially what I'm doing today (on my old server) and then just force clients to use the those version and do their own up/down sampling as appropriate.

Thank you for those ebay links Chris Moore! They really look compelling. But I think 24 and 48 drives is a little too big for me, though they really are inexpensive. Is there maybe a 12 or 16 bay compelling option? I also think I don't need a dual-socket setup; one socket w/ beefy enough CPU should be enough.

Ahh the CSE-836 looks like a 16 (?) bay option. Does it also have the Locate LED feature? Are these bays hot swappable? I think I read that hot swapping does not work well w/ FreeNAS, which is perfectly fine with me ... I can accept a little down time to swap a drive, but identifying which one to replace is really important in the heat of the moment ;)

 

[mikemccand]

OK another problem/constraint -- sorry I missed this on the op -- this rack mount is at the rear of my home theater so noise is a concern, and reading up briefly on the Supermicro cases, it looks like being quiet is not a priority -- they are designed to be racked up in a data center with many others.

 

[IQless]

If noise is a real concern, then maybe an ordinary tower-style case would be a better choice? Something like a Fractal Design R5 or R6?

 

[mikemccand]

Thanks @IQless, those cases do look amazing, but I'm really hoping to stick with rack mount. My current Rosewill rack mount case (https://www.newegg.com/Product/Product.aspx?item=N82E16811147164) is able to keep the drives cool (they are 25C-28C right now) and quite silent after I replaced all fans.

I do love the per-drive LEDs on the Supermicro cases, how easy it is to swap the drives and the redundant power supplies. I wonder, if I get at 24 drive rack mount, but then only populate 10 of the bays, sparsely, so there's plenty of air flow around them, then swap out the fans for quieter ones (that can ramp up if CPU/drive temps get too hot), if that'd be quiet enough. This person did an amazing job making a Supermicro SC846 very quiet: https://www.youtube.com/watch?v=0UjyL6ZiMkI Jealous ;)

He swapped out the power supply for the apparently much quieter Supermicro PWS-920P-SQ

 

[jgreco]

It's holding media, data sets, a growing MySQL DB. It doesn't need to be super high performance.

And with RAIDZ in use, it is never going to be super high performance.

For purposes of this discussion, MySQL is equivalent "block storage".

https://www.ixsystems.com/community...d-why-we-use-mirrors-for-block-storage.44068/

If most of your storage is RAIDZ-friendly stuff and you just have a smallish SQL need, you can always solve the MySQL issue by creating a separate SSD mirror pool for the SQL, which will be amazingly better than anything you'd ever get from any kind of HDD pool.

 

[mikemccand]

Thanks @jgreco -- that's a great read and a great point about MySQL on ZFS. Maybe I'll install a mirror'd pair of SSDs if the MySQL performance becomes a problem later. I'll keep an eye on it ... so far on the RaidZ1 pool MySQL has been acceptable.

I think I'm leaning towards single socket, fairly beefy CPU, 24 drive bay Supermicro case, swapping in quieter power supplies and quieter fans. Supermicro has a crazy number of cases ...

 

[r0nski2000]

@OP
Another solution wrt 4k video, is to not look for 4k video advice on a nas forum...

 

[mikemccand]

Thanks @r0nski2000 that makes total sense ;)