TrueNAS, FreeNAS Certified or DIY build?

[Chainsaw Juggler]

Hi,
having read the great hardware recommendation guide and some more forum posts, I am a bit stumped with my requirements: I want to store video camera recordings of a festival with multiple simultaneous tracks (concerts, discussions etc.). The cameras record to SSDs, and after each event the SSDs are moved from the camera to a workstation which copies the files to a NAS, then the SSDs are formatted to be used again. Each camera produces 90 GB/hour, and to get all the content copied to a NAS we need a sustained write rate of around 300 MB/s (well, at least for 30 minutes). Total required storage size is around 50 TB effective. I'm not looking for good IOPS. Read speed during heavy write is not important.

Looking at the performance numbers for large SATA drives (e.g. WD Red 10 TB), I'll probably have to write to at least 4 of them in parallel to get close to my required write rate. An all-flash solution would work, but I've seen the price and it's not really an option.

What I'm looking at: (feel free to point out any stupid mistakes)

• FreeNAS 2U certified server, Dual 10 GbE, 12 bays of WD Red 10 TB, running with RAID Z2
• TrueNAS X10, Dual 10 GbE, Raid Z2, probably dual power supplies
• DIY build (but AFAICS that rules out buying support from ixSystems, so not really a good option)

Or is such a question something I should ask ixSystems presales?

 

[Chris Moore]

Looking at the performance numbers for large SATA drives (e.g. WD Red 10 TB), I'll probably have to write to at least 4 of them in parallel to get close to my required write rate. An all-flash solution would work, but I've seen the price and it's not really an option.

If I were making a recommendation, I would suggest a 24 drive system with four vdevs of six drives each. That way you have faster throughput to the disks. The quantity of disks is about IO, not about storage capacity.

Or is such a question something I should ask ixSystems presales?

The big difference between your three options.
DIY, you support yourself, by asking for help in the forum.
FreeNAS Certified, you get hardware support from iXsystems, but you are still visiting the forums for answers.
TrueNAS, you have one-stop support directly from iXsystems.

It is just a matter for you to decide how much the support is worth because it does cost. There are plenty of people on the forum that can suggest hardware to you so you can put something together for yourself. We might even be able to suggest a vendor that can put it together for you, but if you want a solution that is built and supported by a vendor, you should call iXsystems.

 

[Chris Moore]

PS. If you went with the 24 drive configuration I suggested, you would only need 6TB drives, not 10TB.

 

[HoneyBadger]

The question to ask here is "what is the cost of your system not working?"

If your business will literally stop if the system does, I would strongly recommend you go with either a TrueNAS build, or a FreeNAS Certified system and have a consultant on retainer.

Also, big fan of the name @Chainsaw Juggler - just don't drop any.

 

[Chainsaw Juggler]

@Chris Moore , thanks a lot for the input.

If I were making a recommendation, I would suggest a 24 drive system with four vdevs of six drives each. That way you have faster throughput to the disks. The quantity of disks is about IO, not about storage capacity.

So if I understood you correctly, IOPS and throughput scale with vdevs. I care more about throughput than IOPS.

The big difference between your three options.
DIY, you support yourself, by asking for help in the forum.
FreeNAS Certified, you get hardware support from iXsystems, but you are still visiting the forums for answers.
TrueNAS, you have one-stop support directly from iXsystems.

It is just a matter for you to decide how much the support is worth because it does cost. There are plenty of people on the forum that can suggest hardware to you so you can put something together for yourself. We might even be able to suggest a vendor that can put it together for you, but if you want a solution that is built and supported by a vendor, you should call iXsystems.

I did call them, and they were very helpful and friendly.

My current plan is to do a DIY build as a backup machine, and get a TrueNAS for the main task. After all, RAID is not backup.

 

[Chainsaw Juggler]

@HoneyBadger, thanks a lot for the input.

The question to ask here is "what is the cost of your system not working?"

If your business will literally stop if the system does, I would strongly recommend you go with either a TrueNAS build, or a FreeNAS Certified system and have a consultant on retainer.

Fortunately, no, it's not a business, so any stoppage would be unfortunate, but not kill us. Data loss, on the other hand, is not something we want to risk.

Also, big fan of the name @Chainsaw Juggler - just don't drop any.

Thanks, I'll try.

 

[Chris Moore]

So if I understood you correctly, IOPS and throughput scale with vdevs. I care more about throughput than IOPS.

IOPS would be higher with 12 mirror vdevs, having four RAIDz2 vdevs gives good, balance of IOPS to throughput. If you're more interested in the capacity, you could go with 3 vdevs of 8 drives. The more drives in a vdev, the more the performance deteriorates.

 

[Yorick]

> I'm not looking for good IOPS.

Except you are. Once your pool starts to fill, all writes are essentially random access.

Four vdevs would get you the throughput and IOPS you need.

It sounds like you will share via SMB. Are those workstations Mac or Windows? I’m asking because the sync write behavior between the two differs. Mac has it on, Windows does not to the best of my knowledge.
If you are going to do sync writes, you’ll want a SLOG. There are inexpensive used options by Intel.
You can also stick with async writes, and force those on the FreeNAS side.

For SMB, plan to use no more than 80% of your NAS capacity to keep throughput high.

 

[Chainsaw Juggler]

For the DIY backup build, I am looking at the following configuration:

• (all refurbished, system is assembled by the seller in Germany)
• Supermicro CSE826 X9DRD-7LN4F-JBOD 19" 2U 12x 3,5" LFF
• 1 x Intel Xeon E5-2640 SR0KR 6C Server Processor 6x 2,50 GHz 15MB Cache
• 16GB Registered ECC DDR3 RAM (2x 8GB DIMM)
• 10 Gigabit Dual Port - 2x 10GbE SFP+ incl. 2x SFP+ 10Gb Fibre Low Profile
• 2x SFF-8087 LSI Broadcom 2308 SAS onboard Raid Controller (IT HBA Mode pass-Through Controller)
• 7 x 3,5" 6TB 7,2k Enterprise Storage 24/7 SAS HDD incl. HotSwap Tray in a RAID Z2 configuration (1 vdev)
• Total price: 1549.58 EUR (+taxes), free shipping

AFAICS the FreeNAS hardware recommendation guide would probably have two issues with this:

• Lack of a dedicated boot drive. I plan to address this with a dedicated small SATA SSD
• Only 16 GB of RAM. I do not plan for any deduplication, but for the given capacity it's a bit on the low end of RAM

Do you think the hardware is reasonable?

 

[Yorick]

Looks okay, I’m just not sure what that 10Gb NIC does for you. A single vdev will give you 100-120MB/s of throughput depending on drive size and rotational speed - enough to saturate a single Gbit link, maybe a little more. It won’t get close to 10Gbit speeds.

Edit: The board already can handle 8 SAS drives; what are the additional controllers for?

 

[Chainsaw Juggler]

IOPS would be higher with 12 mirror vdevs, having four RAIDz2 vdevs gives good, balance of IOPS to throughput. If you're more interested in the capacity, you could go with 3 vdevs of 8 drives. The more drives in a vdev, the more the performance deteriorates.

Ah. That would require a chassis able to accommodate 24 drives. Thanks for the heads-up.
Do you have any pointer to numbers on any sharp cliff in performance for the number of drives per vdev? Looking at the forums, people seem to frown when vdevs with more than 8 drives are discussed.

 

[Chainsaw Juggler]

> I'm not looking for good IOPS.

Except you are. Once your pool starts to fill, all writes are essentially random access.

Oops.

Four vdevs would get you the throughput and IOPS you need.

It sounds like you will share via SMB. Are those workstations Mac or Windows? I’m asking because the sync write behavior between the two differs. Mac has it on, Windows does not to the best of my knowledge.
If you are going to do sync writes, you’ll want a SLOG. There are inexpensive used options by Intel.
You can also stick with async writes, and force those on the FreeNAS side.

Windows.

For SMB, plan to use no more than 80% of your NAS capacity to keep throughput high.

Yes, I noticed that when I filled up one pool of my private FreeNAS build. Up to 90% full, performance was okay, but beyond the 95% mark it wasn't fun anymore. So the plan is to stay constantly below 80% full.

Looks okay, I’m just not sure what that 10Gb NIC does for you. A single vdev will give you 100-120MB/s of throughput depending on drive size and rotational speed - enough to saturate a single Gbit link, maybe a little more. It won’t get close to 10Gbit speeds.

I had been looking at the benchmarks at https://calomel.org/zfs_raid_speed_capacity.html, specifically the variants with a single vdev:

• 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s
• 6x 4TB, raidz2 (raid6), 15.0 TB, w=429MB/s , rw=71MB/s , r=488MB/s

That looked like it won't saturate a 10 Gbit link, but it would result in almost 4 Gbit of network traffic.

 

[Chainsaw Juggler]

The board already can handle 8 SAS drives; what are the additional controllers for?

As far as I understood the seller, the listed controllers are onboard and are probably those handling the 8 SAS drives.

 

[Basil Hendroff]

The big difference between your three options.
DIY, you support yourself, by asking for help in the forum.
FreeNAS Certified, you get hardware support from iXsystems, but you are still visiting the forums for answers.
TrueNAS, you have one-stop support directly from iXsystems.

I didn't know that. Thanks for the clarification.

 

[Chainsaw Juggler]

iXsystems have been very helpful in configuring a TrueNAS X10 according to my needs. Unfortunately, the chosen reseller went AWOL for a few days after initial communication, and during that time we had budget cuts, so in the end I had to make ends meet with some used hardware running FreeNAS.

Thanks again everyone for your help.

 

[blanchet]

There is an excellent FreeNAS server reseller in Germany: Thomas-Krenn

 

[Exhorder]

There is an excellent FreeNAS server reseller in Germany: Thomas-Krenn

But they are not related to iXsystems.
Another server shop, that sells FreeNAS ready servers in Germany is Servershop Bayern