Scale Setup Best Practices

[MurtaghsNAS]

I am new to TrueNAS, and have been trying to absorb enough wisdom to setup my home Scale setup correctly. Before going too far and getting myself up a creek, I would like to present my intended hardware architecture for peer review. Please, if these plans are problematic, stop me before I make a fool of myself.

I plan on 4 pools:
1) I have a 500GB NVME drive I want to partition for the Boot drive and VM storage. My current plan is 50GB for the boot drive. I know that is very generous over the recommended 8GB, but I am planning long-term. 50GB gives plenty of space for Scale to grow, and really, I won't fill the other partition with VMs with my current needs.
2) The rest of the NVME drive I am going to setup as a ZRAID0 partition. It is intended as a high performance partition for virtualization operation/storage. I do realize I am in a high-risk configuration because of my lack of redundancy. I intend on mitigating this by doing user data storage as much as possible in the main storage pool, performing intense backups, and accepting the loss risk. This system is a personal server. My current plans call for Nextcloud, and 2-3 VMs. I can survive a wipeout of this pool. I do have the ability to upgrade this to a mirrored configuration, but that is currently a future plan only.
3) The main storage pool is four 6TB SATA traditional hard drives. This is mostly a media library and personal file storage. I intend on it being ZRAID1.
4) Once everything is stable with this new system, I am intending on decommissioning my current QNAP NAS. I plan on harvesting its four 1TB drives and forming a second ZRAID1 pool. Obviously I can't mirror the entirety of the main storage, but I can duplicate high-value files here to defend against total pool failure on the Main storage. Later on in the life cycle, I will replace these drives with new drives to form a new primary storage pool, and use the current main storage as its backup.

So are there any major issues I missed? Thanks for helping the new guy out.

 

[SnoppyFloppy]

Regarding #1&2, I'm pretty sure you can't partition your TrueNAS boot drive so you are probably better off buying a cheap 120GB SATA SSD for TrueNAS. I think you can get them on amazon for like 20$ or so.

 

[SnoppyFloppy]

Regarding #3&4, I would personally chose a two-way mirror because of the better redundancy and better read performance but that's just my personal preferences.

 

[gbyleveldt]

Regarding #1&2, I'm pretty sure you can't partition your TrueNAS boot drive so you are probably better off buying a cheap 120GB SATA SSD for TrueNAS. I think you can get them on amazon for like 20$ or so.

Actually you can, Ive partitioned a 500GB NVMe into 100GB for Truenas Scale boot and installed. Once all was done I made the 400GB balance a storage pool that I’ve installed all my VMs on.

Theres a Reddit post detailing the process

https://www.reddit.com/r/truenas/comments/lgf75w/scalehowto_split_ssd_during_installation/

 

[beagle]

Yes, it's possible but since it's not officially supported I wouldn't consider it as "Best Practice".

 

[gbyleveldt]

Yes, it's possible but since it's not officially supported I wouldn't consider it as "Best Practice".

That’s true of course. I’m actually curious what the actual drawback is of doing it the way I outlined above.

 

[morganL]

That’s true of course. I’m actually curious what the actual drawback is of doing it the way I outlined above.

The primary issue will be reliability. Flash drives fail faster with heavier write loads. The boot drive will fail at the same time as the virtualization drive. If your drive is good, you may not have any issues, but the probability of zero issues drops. When you do have an issue (e.g a single flash chip fails), you won't have a system to diagnose it.

 

[gbyleveldt]

The primary issue will be reliability. Flash drives fail faster with heavier write loads. The boot drive will fail at the same time as the virtualization drive. If your drive is good, you may not have any issues, but the probability of zero issues drops. When you do have an issue (e.g a single flash chip fails), you won't have a system to diagnose it.

Ok, fair enough. My install will be as reliable as any NVMe based install. As a casual home user I’m happy with the compromise to be honest. Obviously things change when in a production environment. Thank you.

 

[MurtaghsNAS]

Regarding #3&4, I would personally chose a two-way mirror because of the better redundancy and better read performance but that's just my personal preferences.

What type of mirror are you suggesting? A full pool mirror isn't possible due to the pool size differential.

The primary issue will be reliability. Flash drives fail faster with heavier write loads. The boot drive will fail at the same time as the virtualization drive. If your drive is good, you may not have any issues, but the probability of zero issues drops. When you do have an issue (e.g a single flash chip fails), you won't have a system to diagnose it.

Good point. I hadn't taken the logical jump that the wear intensity of the VM partition would thrash the boot drive as well. I want to walk through the failure recovery process. System described above is functional, then the NVME drive critical fails. On discovery, I would attempt to reboot, which would highly indicate the failure. Recovery would be installing new drive (USB or replacement NVME), and reinstalling Scale. I would be able to recover pools 3 and 4 through pool import (barring in-flight errors). The original boot pool is gone, but has already been replaced. Pool 2 is lost, but I already know it is high risk as a ZRAID0 array, and have plans to recover as best as possible. Did I miss anything?

A question that arose while thinking this through: I know you can back up Pool configuration, used to transition between Core and Scale. How often should that backup configuration be generated? Is the minimum working practice to backup on architecture change (pool construction changes, OS update, etc) or is there another factor?

My final comment is an informal vote to find a better solution to the boot drive issue. I am not educated on how we got to where we are, but looking forward, there HAS to be a better solution. The cost of a SATA port (or SAS, etc) for a 20GB boot drive is too high.

 

[Ixian]

SCALE, like CORE, creates an automatic backup of the config db each day and puts it in /var/db/system/configs-* . I use a script to copy the latest backup to my application folder, because I have a regular snapshot schedule set up that also replicates to my backup server (also ZFS).

I run mirrored SSDs for my split-boot setup (64gb boot, the rest my container pool). Works great. Also have a high-endurance Intel 750 I use for SLOG duties that has another partition I offload app cache to (from containers, etc.) as well.

 

[morganL]

Good point. I hadn't taken the logical jump that the wear intensity of the VM partition would thrash the boot drive as well. I want to walk through the failure recovery process. System described above is functional, then the NVME drive critical fails. On discovery, I would attempt to reboot, which would highly indicate the failure. Recovery would be installing new drive (USB or replacement NVME), and reinstalling Scale. I would be able to recover pools 3 and 4 through pool import (barring in-flight errors). The original boot pool is gone, but has already been replaced. Pool 2 is lost, but I already know it is high risk as a ZRAID0 array, and have plans to recover as best as possible. Did I miss anything?

My final comment is an informal vote to find a better solution to the boot drive issue. I am not educated on how we got to where we are, but looking forward, there HAS to be a better solution. The cost of a SATA port (or SAS, etc) for a 20GB boot drive is too high.

That's a reasonable assessment .. it all depends on the value of your data and your time.
For boot, we are increasingly using M.2 boot devices. they have proven to be very reliable.

 

[SnoppyFloppy]

What type of mirror are you suggesting? A full pool mirror isn't possible due to the pool size differential.

I guess you have at least two options should you choose to go with mirrored pool(s). Either you:

1. Create two pools - one pool of the 4 6TB drives and one pool of the 4 1TB drives - each pool will be the ZFS equivalent of raid10 - a striped mirror
2. Or you can add all eight drives to the same pool of mirrored drives as long as you don't mix sizes in the same vdev - in this case you would end up with one pool of consisting of 4 vdevs - 2 6TB mirrored vdevs and 2 1TB mirrored vdevs

So you have choices, and of course you can also just stick to your raidz1 plans, I just don't personally think it adds enough redundancy for my taste.
Here is a good primer on the reasons for choosing mirror over raidzX by the way.

 

[Ixian]

To add my own experience, I became a convert to the mirrored vdev approach when one of my Z2 pools was resilvering due to a disk failure and I then lost a second disk. I was able to recover from there but a bit of a nailbiter because it is A) really, really slow - these were 8TB drives and the whole process (losing the first drive, resilvering, losing the second drive 12 hours later, etc.) took 3 days and B) the entire pool was, practically speaking, unusable during that time. You could access it, of course, but performance was bad, and accessing it also slowed the resilvering even further.

A Z2 or Z3 setup with even larger (12+) drives would be worse than that by far.

Contrast to the mirrored setup I have now on my backup server (10 8tb drives, 5 mirrored vdevs) it's much simpler. You can also upgrade vdev sizes and add space to your overall pool easier. Just fail a drive in one pair, replace it with a larger one, allow the mirror to re-duplicate, then do the same for the original drive in the pair. I upgraded a pair of 4tb drives I had in that same pool to 8 and it only took a couple hours. And performance on the pool, overall, was fine during the process since it wasn't stressing other drives.

Even if you don't need the extra iops it's less stressful and more flexible to go with this kind of setup. Anyone considering RaidZ3 (i.e. you have the quantity of drives/data protection needs to warrent it) should consider mirrors instead, you go from 67% space effeciency to 50% and there's a small statistical chance you'd lose the pool faster if both drives in a mirror fail at the same time but practically speaking you are better off because the recovery process is so much lighter on your pool overall, and faster.