Back in mid 2013, I started working for a post production company and immediately took possession of a brand spanking new Windows 2k12 storage server from a company building and providing 24/7 support. It was a bit of a beast running a 17 drive (Seagate 3TB Barracudas) RAID 6 with a hot spare, 8GB of RAM and dual GigE ports bound in a LAGG-LACP configuration and cooling fans that made me think that an RC jet turbine was running in the room.
The thing was loud and during peak production events, we had 6 editors using Final Cut Pro on machines running running OS X 10.6 to whatever was current at the time connecting back to it via CIFS/SMB.
Those first few months, performance-wise, it was pretty abysmal. Saturate one NIC in just half duplex by doing a media dump of a Sony SxS card onto the NAS and everyone else's workstations would grind to a halt. Save or transcode a video file without announcing that you're going to do it and others holding back, you were pretty much guaranteed to have dropped frames and/or corrupt audio. In fact, in all the monitoring, I never did actually see any combination of in/out that was more than 800Mbit at seemingly half duplex despite there being two NICs in a LAGG and Windows saying it was a 2Gb FDX connection. The support wasn't great either and the best I got out of them was that Gigabit ethernet has overhead which we're running into which causes the performance problems.
On the home front, I'd been playing with FreeNAS on spare hardware and was just switching over to a supermicro X9 with one of the i3's that supports ecc RAM, 6 x 2TB WD green drives running in a Z2 configuration and was getting more comfortable with it. I've since upgraded the drives to 4TB WD reds after drives started failing. After that summer's busy season at the production company, I went home and tested with 3 computers pushing/pulling data to my lowly surplus desktop FreeNAS with WD green drives, onboard realtek and an intel desktop NIC LAGG'd. I managed to see 800Mb on both cards at full duplex.
As the company's busy season started to wind down and I was taking stock of what was going on, I learned that their way of backup and archiving was buying external RAID enclosures and plugging them in via eSATA. The previous admin was banking on this Windows storage server as their archival/backup system.
I proposed and got approval to build a FreeNAS box to act as an offsite backup for the win2k12 machine in early 2014. I don't recall which one but I automatically went for a Supermicro box with a 36 drive capacity. Populated it with a Xeon E5 2609, 32 GB of RAM and 24 x 4TB WD Red drives. This was my first large build and so I made some guesses at how to build it. The purpose would be off site backup storage so it should be quite idle. Two 12 disk RAID-Z2 vdevs in one zpool. (cross my fingers and hope that doesn't come and bite me later in the future).
It took much longer for them to figure out what should be archive and what shouldn't be. In the mean time, that Win2k12 box filled up and (surprise surprise), Seagate 3TB Barracudas are notoriously unreliable. Plus, expanding the RAID6 with those drives from that company cost around $600/drive.
My FreeNAS box still sat there off to the side and the busy season for 2014 was coming soon. So, not wanting a repeat of issues, I created an AFP share and got the editors to dump and work from it. I was happy to see that it worked the way the win2k12 box should have, both NICs in the LAGG were used though I rarely saw the LAGG graph spike above 1Gb but things worked full duplex and both NICs were working. No reports of slowdowns or any other issues.
By early 2015, I finally moved the freenas box out to colocation and then got the order to replace a win2k3 server acting as an application (local windows accounting database) and FTP server. I found a 2u supermicro box with an X10 DRI board in it. Threw in one processor, 16GB of RAM and two 6TB WD Red drives mirrored. A couple of jails. One for windows 7 and the database app and one BSD jail for a development FAMP server that wound up running a web ftp client as well. (Side note, an LGA2011 V3 CPU fits in an R3 socket despite the pin patterns looking different and I now have a spare LGA2011 v2 cpu because of the mixup)
The summer busy season was ramping up along with an expansion and I was tasked again with getting yet another FreeNAS server in. However, the 4u supermicro box didn't come in time and so we temporarily ran things off of the 2u box and this is where things got interesting for me and I learned first hand about how different pieces related to each other. (The win2k12 server has had at least half of its 21 drives replaced due to failure at that point so even with 5TB free, I wasn't about to trust it). Upgrading the 2u box became plan B.
So, with 2GB given to php.ini in the FAMP jail (PHP FTP client hosted in it that is frequently handling large video files), I watched as my mirror got thrashed with constant 40-80MB/s IO. ARC would go as high as 11GB but float around 8GB average and the ARC hit stayed around 91%.
The editors reported network slowdowns periodically with their video playback in Final Cut Pro.
I ordered an extra 16GB of RAM from memoryX and ran out to get five WD Red pro 4TB drives which I setup in a RAID Z1. (This was a workdisk/scratchdisk so the Z2 redundancy wasn't necessary. If anything goes wrong, there should be time to move off of it).
The new RAID Z1 improved things greatly in terms of the experience for editors. No notices of speed bumps and the IO on the drives goes between 10-20MB/s. ARC and ARC hit were still not great but after the RAM arrived and installed, that ARC cache floats around 25GB and ARC hit is 97-99%.
AFP and CIFS are both in use since the editing machines don't have USB3 and it takes a good 20 or so minutes to transfer a full Sony SxS 32GB card as opposed to 5 minutes on the lone PC with USB3 inputs.
Now that there is a lull, I figured on sharing my experiences.
I'm also looking online to see what others have done in video post production environments and reading about SAN and iSCSI. However, given the price and performance of NAS technology, SAN is on its way out.....
However, if I understand SAN and ISCSI correctly, I'm not quite sure how that fits into a workflow where raw footage is shared. I had hooked up my home WinMCE box to FreeNAS with ISCSI for a recorded TV drive. I have been able to to connect more than one host to the same ISCSI target but was advised against doing so since file locking does some funky things. So I'm not quite sure how a SAN fits into a workflow where everyone is sharing the same raw footage.
At any rate, those are just my ramblings and reflection of how things have worked out so far.
The thing was loud and during peak production events, we had 6 editors using Final Cut Pro on machines running running OS X 10.6 to whatever was current at the time connecting back to it via CIFS/SMB.
Those first few months, performance-wise, it was pretty abysmal. Saturate one NIC in just half duplex by doing a media dump of a Sony SxS card onto the NAS and everyone else's workstations would grind to a halt. Save or transcode a video file without announcing that you're going to do it and others holding back, you were pretty much guaranteed to have dropped frames and/or corrupt audio. In fact, in all the monitoring, I never did actually see any combination of in/out that was more than 800Mbit at seemingly half duplex despite there being two NICs in a LAGG and Windows saying it was a 2Gb FDX connection. The support wasn't great either and the best I got out of them was that Gigabit ethernet has overhead which we're running into which causes the performance problems.
On the home front, I'd been playing with FreeNAS on spare hardware and was just switching over to a supermicro X9 with one of the i3's that supports ecc RAM, 6 x 2TB WD green drives running in a Z2 configuration and was getting more comfortable with it. I've since upgraded the drives to 4TB WD reds after drives started failing. After that summer's busy season at the production company, I went home and tested with 3 computers pushing/pulling data to my lowly surplus desktop FreeNAS with WD green drives, onboard realtek and an intel desktop NIC LAGG'd. I managed to see 800Mb on both cards at full duplex.
As the company's busy season started to wind down and I was taking stock of what was going on, I learned that their way of backup and archiving was buying external RAID enclosures and plugging them in via eSATA. The previous admin was banking on this Windows storage server as their archival/backup system.
I proposed and got approval to build a FreeNAS box to act as an offsite backup for the win2k12 machine in early 2014. I don't recall which one but I automatically went for a Supermicro box with a 36 drive capacity. Populated it with a Xeon E5 2609, 32 GB of RAM and 24 x 4TB WD Red drives. This was my first large build and so I made some guesses at how to build it. The purpose would be off site backup storage so it should be quite idle. Two 12 disk RAID-Z2 vdevs in one zpool. (cross my fingers and hope that doesn't come and bite me later in the future).
It took much longer for them to figure out what should be archive and what shouldn't be. In the mean time, that Win2k12 box filled up and (surprise surprise), Seagate 3TB Barracudas are notoriously unreliable. Plus, expanding the RAID6 with those drives from that company cost around $600/drive.
My FreeNAS box still sat there off to the side and the busy season for 2014 was coming soon. So, not wanting a repeat of issues, I created an AFP share and got the editors to dump and work from it. I was happy to see that it worked the way the win2k12 box should have, both NICs in the LAGG were used though I rarely saw the LAGG graph spike above 1Gb but things worked full duplex and both NICs were working. No reports of slowdowns or any other issues.
By early 2015, I finally moved the freenas box out to colocation and then got the order to replace a win2k3 server acting as an application (local windows accounting database) and FTP server. I found a 2u supermicro box with an X10 DRI board in it. Threw in one processor, 16GB of RAM and two 6TB WD Red drives mirrored. A couple of jails. One for windows 7 and the database app and one BSD jail for a development FAMP server that wound up running a web ftp client as well. (Side note, an LGA2011 V3 CPU fits in an R3 socket despite the pin patterns looking different and I now have a spare LGA2011 v2 cpu because of the mixup)
The summer busy season was ramping up along with an expansion and I was tasked again with getting yet another FreeNAS server in. However, the 4u supermicro box didn't come in time and so we temporarily ran things off of the 2u box and this is where things got interesting for me and I learned first hand about how different pieces related to each other. (The win2k12 server has had at least half of its 21 drives replaced due to failure at that point so even with 5TB free, I wasn't about to trust it). Upgrading the 2u box became plan B.
So, with 2GB given to php.ini in the FAMP jail (PHP FTP client hosted in it that is frequently handling large video files), I watched as my mirror got thrashed with constant 40-80MB/s IO. ARC would go as high as 11GB but float around 8GB average and the ARC hit stayed around 91%.
The editors reported network slowdowns periodically with their video playback in Final Cut Pro.
I ordered an extra 16GB of RAM from memoryX and ran out to get five WD Red pro 4TB drives which I setup in a RAID Z1. (This was a workdisk/scratchdisk so the Z2 redundancy wasn't necessary. If anything goes wrong, there should be time to move off of it).
The new RAID Z1 improved things greatly in terms of the experience for editors. No notices of speed bumps and the IO on the drives goes between 10-20MB/s. ARC and ARC hit were still not great but after the RAM arrived and installed, that ARC cache floats around 25GB and ARC hit is 97-99%.
AFP and CIFS are both in use since the editing machines don't have USB3 and it takes a good 20 or so minutes to transfer a full Sony SxS 32GB card as opposed to 5 minutes on the lone PC with USB3 inputs.
Now that there is a lull, I figured on sharing my experiences.
I'm also looking online to see what others have done in video post production environments and reading about SAN and iSCSI. However, given the price and performance of NAS technology, SAN is on its way out.....
However, if I understand SAN and ISCSI correctly, I'm not quite sure how that fits into a workflow where raw footage is shared. I had hooked up my home WinMCE box to FreeNAS with ISCSI for a recorded TV drive. I have been able to to connect more than one host to the same ISCSI target but was advised against doing so since file locking does some funky things. So I'm not quite sure how a SAN fits into a workflow where everyone is sharing the same raw footage.
At any rate, those are just my ramblings and reflection of how things have worked out so far.
Last edited:
