I have a suite of video editors who edit locally then store their raw and cut footage on a NAS. Someone decided to get a regular 12-TB NAS and it died within a month. I won't say who the manufacturer was of the NAS but lets just say it's a cousin of the Bison. Long story short they are asking me to find a NAS solution that will keep up with the work load they have.
The general use case scenario will be multiple Video Editors, around 12, transferring anywhere from 12 to 300GB worth of files at a go. The editors do not edit directly from the NAS so it's primarily used as a data repository. This results in 90% of the reads/writes being sequential with the only random access at the file system being multiple concurrent users. Most of these Video Editors are going to be using Windows Desktops with a couple Macs sprinkled in. Samba support/performance is the primary focus since ideally the video editors will just drag/drop their files to and from a mapped network drive unless someone has a better suggestion. They do have a Windows 2016 Domain Controller if that is of any help but it's network connectivity is limited by comparison. A secondary consideration/priority is the ability for some Linux servers to be able to connect via FTP for uploading relatively small files and then being able to retrieve/browse these via HTTP(S), primarily for Backups.
My current hardware selection is as follows:
- SuperMicro 826BE1C-R920LPB Chassis - x12 3.5" Hot-Swap on a 12GB LSI-SAS Expander Backplane, x2 2.5" Rear Hot Swap (for OS), Redundant 920w PSU
- SuperMicro X10SRL-F - Single Socket 2011-3, x4 DDR 2400Mhz, Gen3 PCIe with four x8 slots off the CPU, IPMI
- Intel E5-1620V4 - 4c/8t, 3.50Ghz base, 10MB L2
- x2 Kingston KVR24R17D4/32MA 2400Mhz DDR4 Registered ECC 32GB so total of 64GB Memory, room to expand to 256GB
- x12 Seagate ST10000NM0206 - 3.5" 10TB Enterprise Capacity 'Helium' 7200-RPM SAS 4kN
- x2 Seagate ST9500620NS - 2.5" 500GB Enterprise Constellation.2 7200-RPM SATA (OS-Drive)
- Chelsio S320E-CXA - 10GBase-CX4 dual-port NIC (The core switch already has CX4 ports, I would have to order SFP+ or 10GbT ports, and a CX4 cable is cheaper)
- LSI (Broadcom/Avago/owner of the week) 9300-4i - 12GB SAS3 HBA
So if I am understanding (or not) all the things I've read from google searches and this forum, here is how I would do the FreeNAS config:
- Two 6-drive RAID-Z2 vdevs in a storage zpool, giving me 80TB capacity
- One 2-Drive vdev in an OS/boot zpool
Does that seem like a correct hardware build and FreeNAS build? Or what recommendations/changes would you make?
I read in a few places that the maximum drive span is 8 drives plus redundancy. I originally thought of doing a 12-Drive RAID-Z2 or Z3 but decided to change it. The mentions I see to this 8-drive limit is pretty dated (2014) so I was just wondering if that is still a consideration or if it's changed.
I read in a few places that the general memory footprint should be an 8GB base plus 1GB per 1TB of storage, but then I also read info that said that's not the case. Is the 64GB I have in the server going to be an issue? Would you recommend going to 128 just for the fun of it or wait and see?
I know samba is very single-threaded performance based so I went with the E5-1620 due to the high CPU clock. Would I be better off going with an E5-1650v4 (6c/12t) or sticking with the current quad-core? There's a marginal difference in CPU clock between the two but the 1650 is twice the price. Not sure if there's a potential bottle neck with just 4 cores.
I did not include a ZIL/SLOG because Samba writes asynchronously by default. Is that the best performance option or would performance be improved by having Samba use synchronous writes and using a pair of Intel 750 400GB NVMe drives as a ZIL/SLOG?
I did not include an L2ARC because the editors all work on different aspects of a project which also means different assets. Most of the time the closest an L2ARC would get to being useful would be when one of the video editors drops a 50GB file on the NAS that another one pulls down to edit locally. Since this is more of a one-off thing and not one guy delivering it to 10 people I didn't see the need for it. I decided it would be best to add an L2ARC after the fact if it is deemed it would be helpful to the overall system performance.
Is Chelsio still the 10GbE NIC of choice? They seem to have a pretty stable and broad support in FreeBSD so I went with them. I know the card I spec'd is a bit old but their Core switch already has a couple unused CX4 ports in it. Cost to add SFP+ or 10GbT would be around $800 for 2 ports. I can buy a 2-meter CX4 cable for $30 and a nic for $80 and be done. If there's a reason I should do this, or if the NIC I selected it known for being unreliable, please let me know.
The other thing is that I believe in an old goverment saying of "Why buy one when you can buy two for twice the price", which leads me into having two of these. If I wanted the secondary to serve as a back-up to the primary, is my go-to going to be rsync? Or does ZFS have some sort of zpool network mirror ability? The intent of the secondary server is to be a 24-hour backup and not a redundant copy. This is because the editors tend to be idiots and squash each other's files all the time. So losing work in a 24-hour span of time that is most likely still sitting on their local machine is fine, but losing more then that is not.
And lastly, is FreeNAS 9.10 capable of working with that hardware list? Or is FreeNAS 11.0 more or less a safe bet at this point?
Thanks for reading my long-winded post and I hope I got most of it right. If you have any recommendations or a different approach please let me know! I want to try and do it right the first time, or do it well enough that I don't have to scrap and rebuild it from scratch.
The general use case scenario will be multiple Video Editors, around 12, transferring anywhere from 12 to 300GB worth of files at a go. The editors do not edit directly from the NAS so it's primarily used as a data repository. This results in 90% of the reads/writes being sequential with the only random access at the file system being multiple concurrent users. Most of these Video Editors are going to be using Windows Desktops with a couple Macs sprinkled in. Samba support/performance is the primary focus since ideally the video editors will just drag/drop their files to and from a mapped network drive unless someone has a better suggestion. They do have a Windows 2016 Domain Controller if that is of any help but it's network connectivity is limited by comparison. A secondary consideration/priority is the ability for some Linux servers to be able to connect via FTP for uploading relatively small files and then being able to retrieve/browse these via HTTP(S), primarily for Backups.
My current hardware selection is as follows:
- SuperMicro 826BE1C-R920LPB Chassis - x12 3.5" Hot-Swap on a 12GB LSI-SAS Expander Backplane, x2 2.5" Rear Hot Swap (for OS), Redundant 920w PSU
- SuperMicro X10SRL-F - Single Socket 2011-3, x4 DDR 2400Mhz, Gen3 PCIe with four x8 slots off the CPU, IPMI
- Intel E5-1620V4 - 4c/8t, 3.50Ghz base, 10MB L2
- x2 Kingston KVR24R17D4/32MA 2400Mhz DDR4 Registered ECC 32GB so total of 64GB Memory, room to expand to 256GB
- x12 Seagate ST10000NM0206 - 3.5" 10TB Enterprise Capacity 'Helium' 7200-RPM SAS 4kN
- x2 Seagate ST9500620NS - 2.5" 500GB Enterprise Constellation.2 7200-RPM SATA (OS-Drive)
- Chelsio S320E-CXA - 10GBase-CX4 dual-port NIC (The core switch already has CX4 ports, I would have to order SFP+ or 10GbT ports, and a CX4 cable is cheaper)
- LSI (Broadcom/Avago/owner of the week) 9300-4i - 12GB SAS3 HBA
So if I am understanding (or not) all the things I've read from google searches and this forum, here is how I would do the FreeNAS config:
- Two 6-drive RAID-Z2 vdevs in a storage zpool, giving me 80TB capacity
- One 2-Drive vdev in an OS/boot zpool
Does that seem like a correct hardware build and FreeNAS build? Or what recommendations/changes would you make?
I read in a few places that the maximum drive span is 8 drives plus redundancy. I originally thought of doing a 12-Drive RAID-Z2 or Z3 but decided to change it. The mentions I see to this 8-drive limit is pretty dated (2014) so I was just wondering if that is still a consideration or if it's changed.
I read in a few places that the general memory footprint should be an 8GB base plus 1GB per 1TB of storage, but then I also read info that said that's not the case. Is the 64GB I have in the server going to be an issue? Would you recommend going to 128 just for the fun of it or wait and see?
I know samba is very single-threaded performance based so I went with the E5-1620 due to the high CPU clock. Would I be better off going with an E5-1650v4 (6c/12t) or sticking with the current quad-core? There's a marginal difference in CPU clock between the two but the 1650 is twice the price. Not sure if there's a potential bottle neck with just 4 cores.
I did not include a ZIL/SLOG because Samba writes asynchronously by default. Is that the best performance option or would performance be improved by having Samba use synchronous writes and using a pair of Intel 750 400GB NVMe drives as a ZIL/SLOG?
I did not include an L2ARC because the editors all work on different aspects of a project which also means different assets. Most of the time the closest an L2ARC would get to being useful would be when one of the video editors drops a 50GB file on the NAS that another one pulls down to edit locally. Since this is more of a one-off thing and not one guy delivering it to 10 people I didn't see the need for it. I decided it would be best to add an L2ARC after the fact if it is deemed it would be helpful to the overall system performance.
Is Chelsio still the 10GbE NIC of choice? They seem to have a pretty stable and broad support in FreeBSD so I went with them. I know the card I spec'd is a bit old but their Core switch already has a couple unused CX4 ports in it. Cost to add SFP+ or 10GbT would be around $800 for 2 ports. I can buy a 2-meter CX4 cable for $30 and a nic for $80 and be done. If there's a reason I should do this, or if the NIC I selected it known for being unreliable, please let me know.
The other thing is that I believe in an old goverment saying of "Why buy one when you can buy two for twice the price", which leads me into having two of these. If I wanted the secondary to serve as a back-up to the primary, is my go-to going to be rsync? Or does ZFS have some sort of zpool network mirror ability? The intent of the secondary server is to be a 24-hour backup and not a redundant copy. This is because the editors tend to be idiots and squash each other's files all the time. So losing work in a 24-hour span of time that is most likely still sitting on their local machine is fine, but losing more then that is not.
And lastly, is FreeNAS 9.10 capable of working with that hardware list? Or is FreeNAS 11.0 more or less a safe bet at this point?
Thanks for reading my long-winded post and I hope I got most of it right. If you have any recommendations or a different approach please let me know! I want to try and do it right the first time, or do it well enough that I don't have to scrap and rebuild it from scratch.
Last edited: