My First FreeNAS Build, am I forgetting something or going wrong?

[W van Elten]

Hi there,

I'm looking to build a Mass storage device for our office.
It will be used to store Autodesk Vault 2017 files (3D drawings), Siemens TIA files, SEE-Electrical drawings and normal office files (doc, pdf, jpg, movies).
Currently it's scattered all over the place (servers, nasses and VM's).
I want to make ONE place where all the files are.
I think it is best to use ZFS RAID-Z2, and think that One pool for Autodesk and One pool for all the other files is the way to go (or should I make one big pool in RAID-Z3?)
1 10Gbit link will be to the switch
1 10Gbit link will be to the Vault Server
1 1Gbit link will connect directly to the EdgeRouter for off-site backup (100/100Mbit fiber connection, or 500/100Mbit fiber connection both sides)

The hardware:
Case: Inter-Tech 19" IPC 4U-4424 24x 2,5/3,5" hotswap (link)
Mobo/cpu: Supermicro X10SDV-4C-7TP4F-O 16x SAS/SATA, 2x 10Gbit (link)
Mem: Crucial CT2K32G4RFD4213, 2x 32GB DDR4 ecc-registered
PSU: Corsair RM850x 80 plus Gold, 8x molex
FreeNAS HDD: 2x (mirror) Crucial MX300 275GB SSD (powerloss-pretection)
ZIL/L2ARC: Samsung 960 Evo 500GB NVME
Storage: 4x Hitachi Deskstar 2TB, 4x Toshiba P300 2TB, 4x WD Purple 2TB, 4x WD Red 2TB
Cables: 4x SFF-8087 to SFF-8643


Questions:
A: Is this set-up OK?
B: 2x a raidz2 pool, or one raidz3 pool?
C: ZIL/L2ARC size ok, or need it to be bigger/smaller
D: OS (FreeNAS) on RAID SSD over-done, or smart?
E: Other remarks?


Thanks in advance!

 

[Ericloewe]

Crucial MX300 275GB SSD (powerloss-pretection)

No, they don't. Not that it's needed for the boot drive, but I do want to clear up that misconception.

Mobo/cpu: Supermicro X10SDV-4C-7TP4F-O 16x SAS/SATA, 2x 10Gbit (link)

Sure.

Mem: Crucial CT2K32G4RFD4213, 2x 32GB DDR4 ecc-registered
PSU: Corsair RM850x 80 plus Gold, 8x molex

Sure.

FreeNAS HDD: 2x (mirror) Crucial MX300 275GB SSD

Overkill, but sure.

ZIL/L2ARC: Samsung 960 Evo 500GB NVME

Getting a bit ahead of ourselves here. tl;dr - no, but we'll come back to this topic.

B: 2x a raidz2 pool, or one raidz3 pool?
C: ZIL/L2ARC size ok, or need it to be bigger/smaller

Have you read cyberjock's guide, linked in my sig? It sounds like you're missing some important information regarding ZFS:

• RAIDZ vdevs really should not be wider than 10-11 drives
• SLOG is probably going to be useless, since it sounds you'll be using regular old file sharing with SMB.
• 500GB is on the large side for L2ARC with just 64GB of RAM. It'd be a good size with 128GB of RAM, which might bring more benefit than L2ARC anyway.

Back to the Samsung 960 Evo:
As L2ARC, it should be okay, but the Pro would be better.
As SLOG, both of them are absolutely incompetent. L2ARC is the exclusive domain of enterprise drives tuned for constant write activity. Even more importantly, neither of them has power loss protection, which makes them less than useless SLOG devices. You might as well turn off sync writes - you have the same risk of data loss, but much more speed.
For SLOG (not that you are likely to need it), there are three devices worth considering: The Intel S3700, Intel 750 and Intel P3700. The S3700 is limited by SATA speeds, the 750 is tuned more for client workloads and the P3700 is the best option.

 

[Ericloewe]

Oh, I forgot something: Why Xeon-D? It's going to be more expensive than Xeon E5, with more limited expandability (especially RAM).

 

[W van Elten]

Thanks for the quick and complete reply!

All the working stations (workstations and notebooks) are running Windows 7 (or 10). Data comes from Autodesk Vault, Siemens TIA or the windows explorer. So I guess a NFS share won't work. And it thus will be a SMB.
So a ZIL (same as SLOG I guess?) will be useless indeed. Thanks!
Moving the ZIL to the mirrored OS drive a solution? (am using a 10Gbit interface and use a lot of small files)

L2ARC is used to cache frequently read files if I stand correct.
I think our system will use a lot of different files each time. So a l2arc is also kind of useless for us?

Regarding the MX300, the specs says it has power-loss protection. (specs)
Is that a different kind of protection they mention?

Also, SSD in Mirror as OS drive is overkill, should I use one drive or a USB stick? And If I use the SSD, can I use that also to install apps/plugins?

Why the Xeon-D?
Its €600,- inc VAT for a 4-core SOC with 2x 10Gbit, KVM, 128GB ecc-DDR4R and an intergrated LSI2116. I think 128GB will be more then enough, as is 16x sas/sata and 4x sata and 2x M2 and 2x 10gbit.
A comparable system will be more expensive I think. not the situation?

 

[SweetAndLow]

Create a single pool not multiple pools. You then create a dataset for each thing you want to separate.

Why the crazy drive mismatches? Just get the same drive so you don't have to deal with different warranties, speeds and firmwares. Make sure to follow the burn in procedure in my signature.

You should also experiment with different vdev layouts, mirrors will be better for your small file workflow. Raidz2/3 will maximize storage.

Sent from my Nexus 5X using Tapatalk

 

[W van Elten]

Create a single pool not multiple pools. You then create a dataset for each thing you want to separate.

Why the crazy drive mismatches? Just get the same drive so you don't have to deal with different warranties, speeds and firmwares. Make sure to follow the burn in procedure in my signature.

You should also experiment with different vdev layouts, mirrors will be better for your small file workflow. Raidz2/3 will maximize storage.

Sent from my Nexus 5X using Tapatalk

Having one single pool with multiple vdevs makes it less reliable isn't it?
One vdev failing in a zpool will make the whole zpool crash. so 2x a vdev of 8 drives is less reliable then 2 pools with a 8 drive vdev per pool. Or am I missing something?
Or wil a vdev of 16 drives with raidz3 be a better option?

The crazy mismatch of the drives is to minimize the chance of production failure failing my pool.
Idea was that one pool with 8 drives consist of 2x each drive in raidz2. Making 2 drives failing not a problem. (Ex Seagate Baracude ST3000DM001 user ;) Had my whole RAID 5 Synology f*cked up because 4 drives failed within 2 days. )

 

[SweetAndLow]

You test your HDD to prevent that. Getting random drives will not help prevent failures.

The whole argument that more vdevs increases the chances of total pool failure is very strange to me. The entire file system was designed to have multiple vdevs. If you want to have multiple pools you should create a single pool for each disk. If that sounds dumb to you that is because it is. A single pool is best and use your backup if it dies.

Sent from my Nexus 5X using Tapatalk

 

[danb35]

The whole argument that more vdevs increases the chances of total pool failure is very strange to me.

I don't know why, because it's absolutely true, as far as it goes. If any vdev in a pool fails, the entire pool fails. Whatever (very small) chance there is of a single eight-disk RAIDZ2 vdev failing is doubled by adding a second eight-disk RAIDZ2 vdev to the pool. If those vdevs are in separate pools, then one pool will survive. In a single pool, though, the entire pool goes.

The counter to this is that, with proper hardware, burn-in testing, and pool layout, the chance of vdev failure is small enough to not worry about--and if the pool does fail, that's why you have a backup (don't you?).

 

[Ericloewe]

So a ZIL (same as SLOG I guess?)

Cyberjock's guide will teach you that the ZIL always exists, but it can be offloaded to a dedicated SLOG device.

Moving the ZIL to the mirrored OS drive a solution?

What? No, that's impossible and makes no sense. No sync writes, no ZIL activity.

L2ARC is used to cache frequently read files if I stand correct.
I think our system will use a lot of different files each time. So a l2arc is also kind of useless for us?

Start off without one. Check the ARC hit ratio under real world conditons. If the real-world data suggests L2ARC would be useful, add it later.

Regarding the MX300, the specs says it has power-loss protection. (specs)
Is that a different kind of protection they mention?

Yes, it's a half-truth. It's only for drive metadata. They're using a few puny caps, whereas real power loss protection involves a couple of supercaps or a big electrolytic or two.

Also, SSD in Mirror as OS drive is overkill, should I use one drive or a USB stick? And If I use the SSD, can I use that also to install apps/plugins?

• Install aps/plugins - no
• Move .system dataset to boot pool, assuming it is mirrored - yes, this might help your performance very slightly
• Use one SSD - you can do that
• Use USB drive - don't bother

Why the Xeon-D?
Its €600,- inc VAT

Ok, that's a decent price.

Having one single pool with multiple vdevs makes it less reliable isn't it?

Slightly. The whole idea is that failure of a vdev is exceedingly unlikely. 2*very unlikely is still veyr unlikely.

Or wil a vdev of 16 drives with raidz3 be a better option?

Definitely not, unless you want an unusable system.

The crazy mismatch of the drives is to minimize the chance of production failure failing my pool.

There are better ways of doing that. Crazy mismatches can cause weird issues, too.

 

[Dice]

Having one single pool with multiple vdevs makes it less reliable isn't it?

I believe there have been some probability/maths discussion on this topic numerous times.
I'd offer an attempt to an alternative formulation that maybe makes additional sense to the views put forward by @danb35 and @SweetAndLow

In a pool the probability for data loss due to drive failures is based on the capacity of each vdev to handle failure. In raidz this is dictated by stripe width.
That means that the likelihood for ZFS to recover from a intra-vdev failure is the same regardless if the pool consists of a single vdev or multiple vdevs.
This is the understanding that makes us calm to have multiple vdevs on a single pool, using ZFS.
Ie, the understanding that assure that every vdev is setup to have enough redundancy to fit our expectations.
So... all we need to do is to maintain every vdev by suppling the cookiemonstah (ZFS) enough cookies to stay fat and happy.

On the other hand - if one does not address vdev issues, there might be some compelling arguments for using multiple pools to minimize data loss.
But this is not improving failure probabilities nor ZFS capacity to recover. The correct way to kindly put it would be:
- multiple pools are better to manage data loss risk in the face of completely ignorant admin. Furthermore, that data would probably be best stored on multiple "put in a box" backups tucked away.

 

[W van Elten]

Thanks all for the answers and info.

So to sum it all up.
All drives to the same drive (WD RED)
1 zpool with 8 disks in a vdev raidz2, having 2 of those (or maybe one to start and expand later on to two vdev's)
1 SSD as OS drive (Is a SATA DOM a solution here?)
L2ARC and ZIL as it is, check real-world conditions to see if a change will help.

What is adviced for apps or maybe a virtual machine storage?

 

[Stux]

All drives to the same drive (WD RED)
1 zpool with 8 disks in a vdev raidz2, having 2 of those (or maybe one to start and expand later on to two vdev's)

Yes. Wise to add the next vdev when you need it. Drives tend to lower price over time... or it gives you time to find a bargain.

1 SSD as OS drive (Is a SATA DOM a solution here?)

If you want. And yes, SATA doms are a fine solution...

SATA drive needs a drive bay + sata port.
SATA dom needs a sata port, but not a drive bay.
Dual USB needs no drive bay or sata port.

Of course a 2.5" drive bay for an SSD could be just some double sided tape on the side of your PSU ;)

L2ARC and ZIL as it is, check real-world conditions to see if a change will help.

Samsung 960 is not suitable for a SLOG (ie what you're calling ZIL).

SLOGs *should* have Power Loss Protection. A SLOG is uses in the event of power loss to replay the synchronous writes which weren't written to the array. It needs to protect from power loss, otherwise its like having a UPS without a battery ;)

If you don't care about the writes... then just throw them away immediately and enjoy the immense speed boost.

 

[W van Elten]

Yes. Wise to add the next vdev when you need it. Drives tend to lower price over time... or it gives you time to find a bargain.



If you want. And yes, SATA doms are a fine solution...

SATA drive needs a drive bay + sata port.
SATA dom needs a sata port, but not a drive bay.
Dual USB needs no drive bay or sata port.

Of course a 2.5" drive bay for an SSD could be just some double sided tape on the side of your PSU ;)



Samsung 960 is not suitable for a SLOG (ie what you're calling ZIL).

SLOGs *should* have Power Loss Protection. A SLOG is uses in the event of power loss to replay the synchronous writes which weren't written to the array. It needs to protect from power loss, otherwise its like having a UPS without a battery ;)

If you don't care about the writes... then just throw them away immediately and enjoy the immense speed boost.

Is there a reason to use a USB instead of a SSD, or vice-versa?
And should it be in mirror, or isn't that needed?
Case has room for 3x 3,5" internal and an internal USB on the mobo.
I wanted to skip the Samsung 960 all together. so have the SLOG on the default setting.
System is of-course connected with a UPS. When there is a power failure, the system is shut down nicely.

Or is a SLOG on an other drive needed? (Intel DC 2,5" s3700 100GB á €170,- or Intel 750 PCIe 400GB á €320,-)

 

[Stux]

Is there a reason to use a USB instead of a SSD, or vice-versa?

And should it be in mirror, or isn't that needed?

A USB drive doesn't use any SATA ports. USB drives are less reliable, and thus should be used in mirror, which FreeNAS will do for you.

SSDs/SATA DOMs are much more reliable, so you don't necessarily *need* to have them in mirrors. Perhaps just having a regularly emailed backup of your config file is all you will require. Pretty much the downtime would be limited to replacing the SSD and uploading the config if your boot SSD were to fail... which is... considerably less likely than a boot USB failing.

Case has room for 3x 3,5" internal and an internal USB on the mobo.
I wanted to skip the Samsung 960 all together. so have the SLOG on the default setting.
System is of-course connected with a UPS. When there is a power failure, the system is shut down nicely.

Or is a SLOG on an other drive needed? (Intel DC 2,5" s3700 100GB á €170,- or Intel 750 PCIe 400GB á €320,-)

You never *need* a SLOG. SLOGs are a performance optimisation if you have a heavy sync write work flow which is bottlenecking you. They are used with NFS and iSCSI servers which are serving virtual devices/filesystems to VM hypervisors... generally.

 

[Chris Moore]

A USB drive doesn't use any SATA ports. USB drives are less reliable, and thus should be used in mirror, which FreeNAS will do for you.

SSDs/SATA DOMs are much more reliable, so you don't necessarily *need* to have them in mirrors. Perhaps just having a regularly emailed backup of your config file is all you will require. Pretty much the downtime would be limited to replacing the SSD and uploading the config if your boot SSD were to fail... which is... considerably less likely than a boot USB failing.

..snip..

I only quoted Stux because he addressed the point I was thinking about and I wanted to pound on it some more...

In my first three NAS builds, I used USB boot devices. Then I had one go bad on me...

The big problem in my view with the USB boot device is that there is no way to run diagnostics on them to even begin to attempt to have some warning about when they are going to fail. In my case, I was at work one day and my wife calls to tell me she can't access the NAS. So I get to come home and spend a big chunk of my evening figuring out what went wrong and fixing it. Add to that, I didn't have a recent backup of my config, which is my own fault, but it being my fault or not didn't make that USB failure any easier to live with.

The point I would like to make, don't use USB. Find a way to use something else, even if it is a spinning disk. You know the old saying about friends don't let friends drink and drive. That can go for USB sticks too... They are fine for a little casual data transfer, but don't get hooked on booting from it... Or something like that...

If you look at the systems in my signature, you will see that my boot pool is a pair of 40 GB laptop hard disk drives and I can get SMART stats from them just like any other disk in my system, so if they are going bad, I can get some warning.

 

[brando56894]

Create a single pool not multiple pools. You then create a dataset for each thing you want to separate.

Sent from my Nexus 5X using Tapatalk

I've learned my lesson about cramming everything in one pool, and it was one that caused me to lose everything. Luckily I had my really important stuff in various off-site places. I now keep my really important stuff in one mirrored pool and my other stuff that I care less about, but have a lot of is in a striped and mirrored pool. I also did the latter so I could expand it quickly.

 

[SweetAndLow]

I've learned my lesson about cramming everything in one pool, and it was one that caused me to lose everything. Luckily I had my really important stuff in various off-site places. I now keep my really important stuff in one mirrored pool and my other stuff that I care less about, but have a lot of is in a striped and mirrored pool. I also did the latter so I could expand it quickly.

So in this configuration if you lose 2 drives you lose all your data. If you would have created a single raidz2 pool and you lost 2 drives you would still have all your data. So explain to me how multiple pools are safer?

Sent from my Nexus 5X using Tapatalk

 

[brando56894]

So in this configuration if you lose 2 drives you lose all your data. If you would have created a single raidz2 pool and you lost 2 drives you would still have all your data. So explain to me how multiple pools are safer?

Sent from my Nexus 5X using Tapatalk

Well it's safer for me, against me, and not necessarily safer against failures. I had somehow screwed up replacing failed drives at least twice before and completely destroyed my pool, so if I happen to destroy my storage pool, my safekeeping with all of my important things is still perfectly fine. Also for me to lose all of my data in my striped mirrored pool, two drives in the same vdev would need to die, which is less likely than two dying in separate vdevs.