vdev Creation Questions for SSDs and JBOD Setup

[Dizzy49]

Total newb here. Final parts for my new server come on Monday.
Dell R740, 2x Xeon Gold 6150, 384GB Ram, HBA330, LSI 9207-8e
2x EMC KTN-STL 15x 3.5" Bay

I'm moving from a Windows/Drivepool setup to TrueNAS Scale. Couple questions...

1) When I install apps, like Emby, where is the data stored? Specifically the database, metadata, cache etc. I know I can specify locations for the libraries. Is it same vdev used to install TrueNAS and I can create/specify a dataset for the Emby data?

2) Related to #1. Is there any reason/benefit to having an "OS Drive" vedv that would be two 250GB 970 EVO Plus SSDs mirrored, and then a larger vdev for other data, VM ISOs, etc. Or better to just create the larger vedv and use datasets for the OS, Apps, ISOs, etc

3) Related to #2. I have two 250GB 970 EVO Plus SSDs, and 10x 870 EVO 1TB SSDs. I was going to have a vdev for the OS (mirrored 250GB SSDs), and then split the 10x 1TB SSDs into two groups of 5 and make pool two vdevs with RAIDZ1, then a Striped vdev of the two RAIDZ1 vdevs. That should give me a decent performance/protection balance. I have two other 1TB SSDs, but I felt 6x in a RAIDZ1 might be pushing it, I would even consider it with mechanical drives, but with SSDs... Maybe?

4) How does RAIDZ work with JBOD? Let's say I have 3x 3TB, 3x 4TB, 2x 10TB, 2x 16TB and 6x 18TB and I wanted protection from 3 disc loss. Throw them all in a RAIDZ3 vdev and TrueNAS will figure it out? Split them up to try to make similar sized vdevs? Group them by size in RAIDZ1/2 vdevs and then create a larger Striped vdev with all of the groups? Other/Better options? I am debating putting the 3TB, 4TB and maybe even the 10TB into the second EMC box to use as an onsite backup. Wouldn't be perfect, but maybe make the vdevs easier?

5) I have quite a bit of data on my drives in the Windows computer. When I create a vdev does it wipe the drives? Is there a way to retain the data? Create a vdev with blank/wipeable drives then expand with drives with data? Or do I need to create a vdev, copy stuff over, then expand the vdev using the drives just just copied from (rinse and repeat)?

 

[Etorix]

2) The boot drive is for… booting. Nothing else. Data, apps and VMs go on other drives. Mirroring the boot drive is of very limited use in home setups.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

4) Throw all drives in a single vdev and you have a 16-wide raidz3 of 16*3 TB: Too wide, and capped by the smallest drive.

Split them up to try to make similar sized vdevs?

Splitting drives into partitions is NOT supported.

Group them by size in RAIDZ1/2 vdevs and then create a larger Striped vdev with all of the groups?

Group by size, but do NOT mix raidz# levels and avoid raidz1. For data safety, it's all raidz2 or all raidz3.

5) I have quite a bit of data on my drives in the Windows computer. When I create a vdev does it wipe the drives? Is there a way to retain the data?

Yes. No.

Create a vdev with blank/wipeable drives then expand with drives with data?

Not possible.

Or do I need to create a vdev, copy stuff over, then expand the vdev using the drives just just copied from (rinse and repeat)?

You cannot "expand" a raidz vdev: Once created, the width is immutable. You may, however, add further vdevs.

 

[Davvo]

1) the system forces you to choose a pool for the apps. It may be your data pool, it may be a different one dedicated to apps and vms running on SSDs. The latter is suggested.

Reading the following resource might help you.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

 

[Dizzy49]

@Etorix
Thank you very much for your reply and explanations. I thought I had an older 128GB or 256GB SSD around here but I may have given it away. I did manage to dig up an 860 Pro 512GB SSD. I'll install TrueNAS on that and back up the config so I can easily recover if the drive dies.

@Davvo
Thank you, that was a helpful read.

Excuse this stupid question, my only experience with RAID was many many years ago, and it was hardware RAID and was an absolute disaster. With RAIDZ1 there is a single parity drive, so you can lose one drive. That much I understand. What I'm not clear on is what actually happens when you lose that one drive. From what I have read I THINK the vdev is down, and needs to be rebuilt after replacing the failed drive. The rebuild is possible because of the parity drive. The parity drive DOES NOT step in and take over for the failed drive.

So if I have 5x drives in RAIDZ1 (rz1-1), and another 5x drives in a RAIDZ1 (rz1-2), and put them in a Striped pool if I lose a drive in rz1-1 that vdev is down until I rebuild it. Since it is part of a striped pool then the pool is also down until the rebuild is complete.

Am I understanding this correctly?

 

[Dizzy49]

Funny, I didn't realize how long ago it was that I used this forum. My signature reminded me. It was that setup that was a complete nightmare. That was like 4 or 5 builds ago :D

 

[Davvo]

Am I understanding this correctly?

Partially. You are right that if you lose a vdev, you lose the entire pool; you are wrong on the RAIDZ# take: you can lose up to # drives in a RAIDZ# layout without harsher consquences than the pool switching to a "degraded" state (and if you lose one more drive, the vdev goes kaputt with little hope of bringing it back).

A read to the following resource might help you remember how ZFS works.

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

For more resources you can look into my signature or in the resource section.

A lot has changed since years ago, have fun with your new machine!

 

[NugentS]

@Etorix
Thank you very much for your reply and explanations. I thought I had an older 128GB or 256GB SSD around here but I may have given it away. I did manage to dig up an 860 Pro 512GB SSD. I'll install TrueNAS on that and back up the config so I can easily recover if the drive dies.

@Davvo
Thank you, that was a helpful read.

Excuse this stupid question, my only experience with RAID was many many years ago, and it was hardware RAID and was an absolute disaster. With RAIDZ1 there is a single parity drive, so you can lose one drive. That much I understand. What I'm not clear on is what actually happens when you lose that one drive. From what I have read I THINK the vdev is down, and needs to be rebuilt after replacing the failed drive. The rebuild is possible because of the parity drive. The parity drive DOES NOT step in and take over for the failed drive.

So if I have 5x drives in RAIDZ1 (rz1-1), and another 5x drives in a RAIDZ1 (rz1-2), and put them in a Striped pool if I lose a drive in rz1-1 that vdev is down until I rebuild it. Since it is part of a striped pool then the pool is also down until the rebuild is complete.

Am I understanding this correctly?

No
If you have a Z1 vdev - you can lose a single disk in the vdev which will still work - the pool will be degraded, but still accessible. Lose another disk in that vdev and the whole pool is toast.

 

[Davvo]

Additionally, losing a parity drive doesn't influence the VDEV's performance, just its resiliency.

Basically, it changes a colour and starts screaming alerts.

 

[Dizzy49]

Great, I think I'm clear on what I'm going to do with the SSDs for the important storage/backup.

3 vdevs with 5x 1TB SSD RAIDZ2
Then 1 striped pool with those 3 vdevs.

My JBOD is going to be a whole other issue... Need to see if I can find some good pricing on 16-18TB drives.