Boot ssd + 4x4TB Raidz2 + 2x ssd cache acceleration (read/write)

[saNeurT]

Hi everyone, I recently bought a TrueNAS Mini X+ and I started to research about the best config for Raidz2, ssd cache acceleration, metadata on flash and fusion pools.
This article about raidz was fantastic, but I still need help.

TrueNas Mini X+ hardware has:

• x5 3.5" bays
• x2 2.5" bays
• 64GB ram
• 8 core, 2.2ghz cpu

My system will have a lot of read/write actions and I mainly look for a mix of data reliability and space efficiency. It will manage different applications that will eat a lot if resources and it will manage around ~8 jails.

I believe to have enough performance thanks to the octacore cpu and ram.
After my research, based on my budget and scope, I'm think that my ideal setup should have:

• TrueNas installed in a ssd
• 4x4TB WD Red Plus CMR HD

I'm still in doubt regarding the other two free bays. I was reading about the possibility of doing ssd cache acceleration... Does it make sense to use the two remaining 2.5" bays by adding x2 ssd (one for read cache acceleration, and the other for write)?

Or is it better to make use of OpenZFS 2.0 and use one 2.5 ssd for metadata on flash and the other one for fusion pools?

What's the difference? What size should I look for the x2 ssd, If I will use 4x4TB Raidz2? 128gb? 512?

In both cases, since I will use the two 2.5", is it smart to buy a 2.5" to 3.5" adapter and mount a ssd on it (used for booting truenas), and place it in the remaining free 3.5" bay?

Thanks in advance for the help!

 

[sretalla]

use one 2.5 ssd for metadata on flash and the other one for fusion pools?

The metadata VDEV is integral to the pool, so a single drive will kill your pool if lost.

 

[saNeurT]

The metadata VDEV is integral to the pool, so a single drive will kill your pool if lost.

Thanks for your answer. What's the best config for me given the fact that I have x5 3.5" bays and x2 2.5" bays?

I mainly look for a mix of data reliability and space efficiency.

 

[sretalla]

A mirrored pool of the 2 SSDs for jails/VMs and system dataset along with jail app config and databases.

The rest in RAIDZ2 as a pool for your data, to be mounted into jails or shared via NFS/SMB.

You could use the third ssd as a metadata only L2ARC if you need to speed up file browsing and/or handling of lots of very small files in the pool.

Boot from a USB to SSD adapter (with a 4th SSD) is the option I would recommend.

 

[saNeurT]

A mirrored pool of the 2 SSDs for jails/VMs and system dataset along with jail app config and databases.

The rest in RAIDZ2 as a pool for your data, to be mounted into jails or shared via NFS/SMB.

You could use the third ssd as a metadata only L2ARC if you need to speed up file browsing and/or handling of lots of very small files in the pool.

Boot from a USB to SSD adapter (with a 4th SSD) is the option I would recommend.

Sorry I'm really new to this space, so I need to make sure that I understood right:

• x2 ssd in the 2.5" bays that will contain jails, system dataset, jail app config and databases
• x4 hdd in the 3.5" bays that will contain data. Raidz2.
• x1 ssd for l2arc
• x1 ssd for boot, mounted via an usb adapter

I have some questions:

1. Regardening x2 ssd mirrored pool, can I set it via the TrueNas UI?
2. Regarding raidz2 pool data, what's the difference between "mounted into jails or shared via NFS/SMB"?
3. Is it really necessary l2arc in my case? Does it give a good performance boost?

Regarding the setup, I'm looking to build it in the most secure way.

 

[sretalla]

Regardening x2 ssd mirrored pool, can I set it via the TrueNas UI?

Yes

what's the difference between "mounted into jails or shared via NFS/SMB"?

Datasets mounted into jails are directly available to the jail like locally attached disk (so-called nullfs mounts). You can't do that with external systems, so would use network sharing protocols like SMB or NFS. For VMs, you can't use nullfs mounts, so NFS is usually best to get storage from the NAS pool into the VM.

Is it really necessary l2arc in my case? Does it give a good performance boost?

Necessary? absolutely not. Performance boost? maybe, depending on how you use it.

Regarding the setup, I'm looking to build it in the most secure way.

Secure could be interpreted in terms of data safety or IT security... not the same thing, so I assume you mean data safety.
RAIDZ2 will allow you to lose any 2 of the 4 spinning disks without losing any data.
Mirrored SSDs will allow one SSD to be lost with no loss of data.
L2ARC poses no risk of data loss, it's only a copy of data you have on the spinning disks.
The boot drive contains only the appliance/OS files (config backups are automatically saved daily to your system dataset... which I recommend to put on the mirrored SSDs) You can take a config backup at any time and restore it to a new SSD with ease (and no data lost from your pools).

If you meant IT security, perhaps you're going to be considering ZFS native encryption. Please do a bit of reading about that as it brings security together with complications and more administration tasks (which can end up losing your data if you don't do the right things).

 

[saNeurT]

Yes

Datasets mounted into jails are directly available to the jail like locally attached disk (so-called nullfs mounts). You can't do that with external systems, so would use network sharing protocols like SMB or NFS. For VMs, you can't use nullfs mounts, so NFS is usually best to get storage from the NAS pool into the VM.

Necessary? absolutely not. Performance boost? maybe, depending on how you use it.

Secure could be interpreted in terms of data safety or IT security... not the same thing, so I assume you mean data safety.
RAIDZ2 will allow you to lose any 2 of the 4 spinning disks without losing any data.
Mirrored SSDs will allow one SSD to be lost with no loss of data.
L2ARC poses no risk of data loss, it's only a copy of data you have on the spinning disks.
The boot drive contains only the appliance/OS files (config backups are automatically saved daily to your system dataset... which I recommend to put on the mirrored SSDs) You can take a config backup at any time and restore it to a new SSD with ease (and no data lost from your pools).

If you meant IT security, perhaps you're going to be considering ZFS native encryption. Please do a bit of reading about that as it brings security together with complications and more administration tasks (which can end up losing your data if you don't do the right things).

What capacity do you recommend for the three SSDs? (One will be used just for TrueNas boot, the other two for will contain jails, system dataset, jail app config and databases).

My idea was two buy Crucial BX500 120 GB for booting TrueNas, and two 250GB SSDs for jails, datasets, configs, etc.
These are the 250GB SSDs that I'm looking:

• Western Digital 250GB WD Blue 3D NAND Internal PC SSD
• Crucial BX500 240GB 3D NAND
• SanDisk - SDSSDH3-250G-G25 Ultra 3D NAND 250GB Internal SSD
• SAMSUNG 870 EVO 250GB 2.5 Inch SATA III Internal SSD

What are the best ones? I read about ZIL/Power Loss Protection/SLOG but I can't understand if these SSDs are ok or if I should check for other products. Can you help me?

Also, does it make sense to buy x2 250GB SSDs or should I opt for the Crucial BX500 120GB?

Thanks in advance!

 

[Newfoundland.Republic]

For my boot drives I have 2 x Kingston A400 120 GB mirrored SSDs. They were cheap at under CND$20 each. 120GB is much more than needed for boot but you shouldn't use the boot drives for anything else. The difference is that I have room and SATA ports to do so in my server.

If I had room in my server - and I have been thinking about replacing my "Small_n_Slow Pool" spinning rust drives with SSDs just for VMs (shared by NFS to my ESXi server) - I'd be looking for 1 or 2 GB SSDs x 3 in a RAIDZ1 config. But that is just my setup.

 

[sretalla]

• Western Digital 250GB WD Blue 3D NAND Internal PC SSD
• Crucial BX500 240GB 3D NAND
• SanDisk - SDSSDH3-250G-G25 Ultra 3D NAND 250GB Internal SSD

I would stay away from these as at least the WD blue ones have a controller that doesn't play well wit FreeBSD and TRIM.

The samsung drive might be OK, but look closely at the TBW values for any drive you consider for L2ARC as all writes to your pool will go there too.

I would not even consider for a moment using any of those for SLOG. (look at the threads discussing the best slog drives... and think a lot harder about your pool layout as SLOG and RAIDZ2 are counter-indicated).

We haven't discussed you having any need for SLOG in this thread...