Multi-Purpose System need input

[zdenkers]

Ok so I've been doing some mild digging around a few places and decided that I think freenas will work well for what I want. However I need to know for sure. So I've compiled a tentative parts list. First you probably want to know the purpose of the machine in question. I'd like it to perform a few different functions: Plex server(includes housing media, 4k transcoding as needed, and media database.), surveilance zoneminder server(includes recording, motion detecting, and live feed viewing as needed), a minecraft server housed in a ramdisk, and a more or less photo/video dumping ground. Each of these tasks will be divided into their own docker containers with the exception of the photo/video dumping ground is obviously just network storage use and need not live in a separate vm. Now onto the hardware:

MOBO: gigabyte mz32-ar0 https://www.gigabyte.com/Server-Motherboard/MZ32-AR0-rev-10/sp#sp

CPU: 1x epyc 7402p https://www.newegg.com/p/N82E16819113592

Memory: 8x samsung 32gb ecc ddr4-2666mhz 2gx4 https://www.superbiiz.com/detail.php?name=D42632G4S2&c=CJ&cjevent=6d458db8d8ba11e9827f00140a1c0e0d

High speed storage for software/plex database: 1TB 970 evo plus https://www.amazon.com/dp/B07MFZY2F2/?

Raid drives: 8x 8TB ironwolf pro 7200rpm drives (raidz2) https://www.amazon.com/dp/B06XX4HBY8/

Does this seem sufficient from the freenas side of things? are there any concerns with hardware compatibility? any concerns with the number of vms or their configuration being problematic? Any hardware change reccomendations? Thanks in advance

 

[PhiloEpisteme]

Plex server(includes housing media, 4k transcoding as needed, and media database.)

You may have already done this, but check Plex's passmark score suggestions for 4k transcoding. That should inform your CPU choice quite nicely.

CPU: 1x epyc 7402p https://www.newegg.com/p/N82E16819113592

I'm curious if there is a cheaper CPU you could go with that would give you the performance you need? Again, I would rely on the passmark score. Is it possible to have your media already in the correct 4k format so you don't have to transcode?

Memory: 8x samsung 32gb ecc ddr4-2666mhz 2gx4 https://www.superbiiz.com/detail.php?name=D42632G4S2&c=CJ&cjevent=6d458db8d8ba11e9827f00140a1c0e0d

I would've suggested go with the largest modules your board supports so you can upgrade in the future but your board supports 16 modules so I think you're fine here.

High speed storage for software/plex database: 1TB 970 evo plus https://www.amazon.com/dp/B07MFZY2F2/?

You're not really getting any benefits from using FreeNAS if you go with a single disk. What some folks do in these kinds of situations is use a pool composed of many stripped mirror vdevs to get good performance and use a fast SSD in front as a SLOG to get better sync write speeds. With your high memory system you should get good read speeds.

Raid drives: 8x 8TB ironwolf pro 7200rpm drives (raidz2) https://www.amazon.com/dp/B06XX4HBY8/

I would suggest you consider the WD 5400rpm drives. Because of how these pools work you're not really gaining much with 7200rpm other than added heat. Not a hard rule, of course, but something to think about.

Does this seem sufficient from the freenas side of things? are there any concerns with hardware compatibility? any concerns with the number of vms or their configuration being problematic? Any hardware change reccomendations? Thanks in advance

You're got a beast here it looks like. I would guess it is plenty for what you're proposing. I didn't go through and check all of the compatibility bits though; for example I didn't see that the board supports ECC memory, but I assume it does.

 

[zdenkers]

The single disk on the m.2 drive really for more miscellaneous system things. It will far exceed the read write speeds of the raid based on my understanding. Correct me if that's not the case.

The biggest question i have is about the 5400 rpm drives. My understanding was that the array would be limited most by raw iops of a single drive. So my thought was to go with a faster drive. Also the pro model drives have better longer warranties. I hope to make this system go 10+ years before it needs to be replaced. Would the 5400 drives really perform the same?

 

[PhiloEpisteme]

I hope to make this system go 10+ years before it needs to be replaced. Would the 5400 drives really perform the same?

If you stripe mirror vdevs you'll get better IOPS than using RAIDZ2. I doubt you'll have any system go that long without needing disk replacements regardless of the type you use. If you think you can cool them effectively you could go with them, but my bet is you'll see bigger IOPS gains by changing your pool layout.

 

[zdenkers]

I figured i would definitely have to replace drives but with 5 year warranties that's hopefully only once each. Vs the 5400 drives are 3 year warranty so effectively twice each to replace.

I like the raidz2 for the ease of replacement on drives and the low loss of overall storage only 25% vs 50% on a mirrored layout.

 

[zdenkers]

As for cooling i intend for this to be a 4u solution so i am hopeful that cooling wont be an issue.

 

[drinking12many]

I think it looks like a pretty good system if not complete overkill, but hey its your money :). (I wish I could afford it) I have run/am running Freenas on much lesser equipment and I have had few to no issues with it from a hardware/basic functionality perspective. I almost never have to transcode 4K content but even my current FX-8320 890 chipset based system CPU has never really been an issue. Generally, I have run into a lack of ram for everything I am trying to do first (only have 24GB currently). I have 1 VM, 6-7 Jails (including plex, radarr,sonarr, etc), also use it as RDM/Datastore in vmware using ISCSI.

I do agree with the above though that 7200 RPMs probably aren't worth the extra heat unless you just want to long as you have the cooling putting a SLOG in front would do more. As far as hardware failures my box is kinda a mess of drives and even with 7200 RPM drives I have yet to have a drive die in it in over 3 years. the only thing I had die was a used HBA which lost on of the connectors (I believe may have been heat-related and put a standalone AC in that room) Honestly though it's your money but I think even a cheaper threadripper or 3000 series (with an ecc board if you want it) would be way more than enough for your needs and still be relevant in 3-4 years.

 

[zdenkers]

So i can't go to a 3000 series and I would do a threadripper if the Zen 2 threadrippers were out. But those are honestly about the same pricing as epyc cpus anyway. The reason i need to go to epyc/thread ripper is because of the exact issue you're bumping into memory constraints. Ideally i would do about 156gb of memory but there isnt really a 156gb configuration lol. So 256 will do fine and give me plenty of headroom. Mostly because i intend to use ram disks for at least one VM. And ryzen doesnt support that much memory.

As far as being relevent in 3-4 years i need this machine to be maintainable for 10ish to justify the cost. Cutting corners and staying relevent for 5 years means spending more money in the long run. These epyc cpus are so cost effective right now that it almost doesn't make sense to even buy desktop hardware for desktops lol. I just dont see how a 3000 series could keep up for 10 years. Im also assuming that 8k content will become available during the lifetime of this machine.

Basically i need more ram than ryzen can support. And threadrippers Zen 2 isnt out yet and will likely be about the same pricing as epyc anyway.

 

[drinking12many]

If that's your goal your probably there. My current mobo supports up to 32GB, but I have an older E5 V2 server on the way that technically supports up to 1TB with the right DIMMS, but I am not sure my use case will every require me going over 64GB which is what I bought for it (if I find it cheap enough on EBAY who knows). Its supposed to be here today. I never watch more than 2 4K streams at once and my TVs play it natively so transcoding wasn't a concern. 8K will definitely be more common by 10 years, but honestly I only see mild improvements on my 55 inch I am in no hurry to go to 8K.

 

[zdenkers]

Does anyone know if gigabyte server mobos generally have good sata controllers for freenas? That's probably my biggest concern here is compatibility. Don't really want to get all the hardware and then can't get freenas running because of compatibility.

And then in terms of performance how much difference will there be between the 5400 drives and 7200 drives in this config?

 

[drinking12many]

well you didn't mention if you are putting a 10Gb nic in it (looks like dual 1Gb links), but even a single 5400 RPM hard drive can saturate a 1Gb link. I think even with 8 5400RPM drives you would probably be close to maxing a singe 10Gb link. Power savings on average (very general here) is usually like 3-4 watts lower per drive for 5400 vs 7200. Yes you would get slightly more IOPS etc out of 7200 RPM drives but its pretty minimal considering the difference even the crappiest SSD could deliver in that regard.

As far as SATA controllers those are delivered by the AMD chipsets/CPU so I wouldnt worry much about those.

 

[zdenkers]

I do intend to have a 10gbe nic. But if 5400s are possibly going to saturate that i will probably go with 5400s and consider an extended warranty.

 

[zdenkers]

Although currently the 7200 drives are actually 10 bucks cheaper lol.

 

[joeinaz]

Did you decide on a specific case?

 

[zdenkers]

Rosehill 4u rsv-l4000 https://www.amazon.com/dp/B0056OUTBK/

 

[PhiloEpisteme]

Rosehill 4u rsv-l4000 https://www.amazon.com/dp/B0056OUTBK/

I can't speak to the noise level difference but there are plenty of 4u cases which hold way more than drives than that case; 24 drives is easily achievable. You may not need that many bays, but you may want to have a spare bay or two so you can burn in new drives etc without having to remove a drive. Many used supermicro cases in the 3u format hold 12 disks, that may be an option you could consider. Used on ebay they often come with the backplane, fans, and dual PSUs. I would say that budget and noise are the factors there; if you can swing ~300 for the case + drives and have a place you can put it without worrying about the noise you may prefer a different case. In general for chassis you can get a lot more bang for your buck if you go used.

 

[zdenkers]

The rosewill case is 100 bucks and noise isnt a problem as this will be going in a concrete vault.

If i can find a 4u case with more drive cages and as good of airflow i would probably consider it if the price is right.

 

[joeinaz]

I have 3 different tower cases that I use(d). I just recently sold my Lian-Li PC-A76 12 disk tower on eBay. I have a 10 disk mid tower (on wheels) that may go on eBay and I have an Antec 900 mid tower with 12 disks internally. Does your case have to rackmount?

 

[zdenkers]

Yes it needs to be rack mount.

 

[PhiloEpisteme]

If i can find a 4u case with more drive cages and as good of airflow i would probably consider it if the price is right.

Are you purchasing a PSU for this build as well? If so I figure you'll probably spend ~$100 for a PSU as well. If you can go rackmount the 3U supermicro CSE836 series can be had for cheap on ebay and typically have 12 bays for ~$300. That is an additional $100 for more drive bays, redundant PSUs, and a likely better built chassis than the one you posted. In fact, here is one for under $300 which even has a board (though it is quite old and perhaps you'll want to not use it). Here is another with a direct-attach backplane that can be used with 1 HBA if your motherboard has plenty of SATA ports; else you can pick up 1 cheap expander.