BUILD Supermicro 24 Bay, 10 Gbe, Server Build

[orionsbelt0]

Hey! Im new here but excited to join the FreeNAS community, there seems to be ALOT of really excellent documentation and support on this platform.

I do professional video as a rapidly growing profession (among a few others) and have been getting busier and busier over the past few months and have outgrown my current storage system.

My current workflow is to ingest a project (usually 500GB - 1TB) onto my internal 1TB SSD while backing up and archiving to a Drobo 5N. The pitfalls of this are that my Drobo is full and is not fast enough to edit directly off of so to pull a project out of archive means transferring it back to a working drive. This is a pain for many many reasons.

What I hope for in my new build:
A scalable solution that I can grow with up to around 100 TB of useable space.
A storage solution that is fast enough to edit directly over the network (10 Gbe, currently a single editor)
A machine that is capable of transcoding 4K media files into proxy files.

My hardware plan is to purchase a barebones partially populated system from Iron Systems with the specs below.
Supermicro SuperServer 6048R-E1CR24L - 24 Bay, Dual 10 gbe, Dual Xeon, Redundant 920W Power Supply
1 - Xeon E5-2637 V4
4 - 16GB DDR4 2133MHz ECC REG
6 - 6TB SATA 6GB/s 7200RPM 128MB (Toshiba Enterprise Drives)
Add a X540 T2 10 Gbe NIC to my editing machine and do a direct connection


The hardware above would get me started and house my current media and give me some room to grow.
In the future I plan to add the following as my bussiness potentially grows, I am aware that if the hardware goes obsolete it will be impossible to find. I hope to upgrade within 1 - 1.5 years.

Stage 2 Upgrades
Additional E5-2637 V4
Additional Matching 16 GB RAM DIMMS
Additional Hard Drives (Ideally in sets of 6)
Add a ZIL or L2ARC if neccesary for throughput
Add a 10 Gbe switch when I am ready for additional editors

My plan is to
Set all 6 initial hard drives to a single VDEV in RAIDZ2 and expand the pool later by adding additional 6 Disk RAIDZ2 VDEVs
Run Adobe Media Encoder within a Windows VM inside a Jail, I will need to play with the appropriate hardware allocation as to not bottleneck FreeNAS

I would like to purchase a UPS once I measure the load draw, I would hope to have about 15-30 min of run time to provide a proper shutdown upon power failure.

Backup Plan for server.... I have a large (64TB) collection of misc. bare drives that I will manually backup files to and keep offsite. Not perfect but better than nothing until I can build a second server to backup to or get a fast enough network connection for offsite.


Any thoughts would be appreciated! Am I missing anything?

 

[jgreco]

It's nice to see someone present what appears to be a fairly well-reasoned, well-designed system. Congrats.

After drive spinup, your maximum load should be around 250 watts with the single CPU and six disks, probably idling more around 175 (and I'd love to know how close I am if you measure it). With two CPU's and 24 drives, more like 550 watts and idling maybe around 400?

Your system as designed with 24 x 6TB in 4x RAIDZ2 will only give you a 96TB pool, of which the 80% rule would suggest only gives you around 77TB of *usable* space. You might wish to consider 8TB drives instead. Or possibly the 36 bay version of that server, which if memory serves does exist.

With the server grade boards, it is fairly unusual for stuff to be "impossible to find." Obsolescence in this arena is totally different than it is in the consumer computer world.

 

[orionsbelt0]

Thanks for your input and compliments! Oh yes I have been sorting through all of the threads of people trying to build with strange hardware for many weeks now.

I will for sure measure the load and post my findings!

Im pretty well tapped on my initial investment on the front end of this but will look at and weight the potential of going with 8TB drives.

On expanding my ZPool in the future.... Do VDEVs have to match? Could my next VDEV add be something weird like 10 - 8 TB drives in a RAIDZ2?
Also I know all of the drives in a VDEV are recommended to be the same size, but what about matching drives? Can I use 3 - WD 6TB and 3 - Seagate 6TB in the same 6 Disk RAIDZ2 VDEV?

Thanks for your help! I will continue to engage the community as my adventure continues.

 

[jgreco]

I am a strong proponent of heterogeneous pools (search term: heterogeneous in the forum search). The drives should be the same size and ideally the same speed; mixing things will cause it to perform as poorly as the slowest/smallest/whatever.

We do not suggest using differing numbers of drives in future vdevs. Six is a great number. Mixing vdevs with different numbers of drives creates many performance challenges.

It is also "frowned upon" to use different drive SIZES in additional vdevs, but primarily because it may lead to unequal pool IOPS loading over time - a larger vdev will see a larger percent of the workload. All things considered, this is not terrible, and if it is convenient to create a 6-drive 6TB-disk vdev today, and in three months you want to add a 6-drive 8TB-disk or 10TB-disk vdev because you need the space, this is not going to be the end of the world. ZFS will intelligently cope with the situation as well as can reasonably be expected. It would probably be ugly in the long term if you ended up with a vdev of 1TB disks, and another of 2TB disks, and another of 6TB disks, and another of 12TB disks. You'd get very weird IOPS distribution and it might be "frustrating" that a 24 disk system was acting a lot slower (because so much of the load is ending up on the 12TB-disk vdev).

 

[orionsbelt0]

Ahhhh.. That makes perfect sense. Essentially trying not to create bottle necks and maintain performance. Yeah I definetly wont do something crazy like 1TB VDEVs or mirrored. If funds allow when I upgrade I may do 8TB VDEVs in the future though.

Thanks for your help Ill update this thread once the machine comes in!

 

[gdreade]

Thanks for your help Ill update this thread once the machine comes in!

So, did you ever build this system? How did it turn out? I've got a very similar workload that I've got to handle and am looking at the Supermico 6048R-E1CR24L. (The video editing it happening on the users' workstations, so this is more a case of online archival data.)