SOLVED Best way to use my available disks

[bigjohns97]

First time user here and wondering if maybe I should rethink the way I have my disks deployed.

Current setup
2x nvme mirror - System
1x nvme + 1x ssd mirror - Apps
7x 10tb WD Red - 3 mirror vdev + hot spare

I have the suspicion that my SSD is failing and I am wondering if I should even replace it. I would much rather just have the 3x nvme and the 7x mech drives but I had some questions about fault scenarios being a new user to ZFS and TrueNAS.

Should I really be wasting the 2x nvme on the system setup, it almost seems like all of the config and work goes into the Apps so I should really move my mirror to Apps and then just use a single nvme for System?

Because if System fails I just replace and reinstall truenas and then discover both pools and then all of my data and apps, containers, vm's all come back and work just fine?

Or does it not work like that?

I would possibly pick up a couple of SSD's and use that for a new mirror for system, and then I would have the following.

2x ssd mirror - System
2x nvme mirror - Apps
2x 10tb WD Red - 3 mirror vdev + hot spare (and maybe even throw another nvme as metadata?)

Anyways appreciate the consideration, thanks in advance.

 

[Ericloewe]

Should I really be wasting the 2x nvme on the system setup, it almost seems like all of the config and work goes into the Apps so I should really move my mirror to Apps and then just use a single nvme for System?

The boot pool is, by far, the most disposable part of the system. So yes, mirroring the boot pool is overkill in many scenarios. Mirroring the boot pool to the detriment of any other data is massively counterproductive.

Because if System fails I just replace and reinstall truenas and then discover both pools and then all of my data and apps, containers, vm's all come back and work just fine?

It's a lot easier if you keep backups of the config database, as you should be doing. There's the edge case of encrypted datasets - if you have any, be sure to have backups of the keys.

(and maybe even throw another nvme as metadata?)

Metadata vdevs must be reliable, losing one means losing the pool. You can't just add a single disk as a metadata vdev, that's a recipe for disaster.

2x nvme mirror - System
1x nvme + 1x ssd mirror - Apps

NVMe devices are SSDs. If you're talking about SATA SSDs vs. NVMe SSDs, please be explicit and precise, as it makes things easier for everyone.

 

[bigjohns97]

The boot pool is, by far, the most disposable part of the system. So yes, mirroring the boot pool is overkill in many scenarios. Mirroring the boot pool to the detriment of any other data is massively counterproductive.

It's a lot easier if you keep backups of the config database, as you should be doing. There's the edge case of encrypted datasets - if you have any, be sure to have backups of the keys.

Metadata vdevs must be reliable, losing one means losing the pool. You can't just add a single disk as a metadata vdev, that's a recipe for disaster.

NVMe devices are SSDs. If you're talking about SATA SSDs vs. NVMe SSDs, please be explicit and precise, as it makes things easier for everyone.

Thanks for the input and yes I was referring to sata ssd above.

It sounds like I need to not worry about the metadata vdev until the problem arises and just reinstall using only a single nvme and reconfigure my apps pool to use dual mirrored nvme and probably not worry about replacing the failing sata ssd.

 

[sretalla]

Better to think of using the NVME as a metadata only L2ARC (which can easily be tested and removed if not beneficial, and doesn't need to be redundant as it's only a cache). Usually recommended to have a lot of RAM for that, but 64GB is OK, so why not.

lots to read about that here:

ZFS "ARC" doesn't seem that smart...

Without yet resorting to adding an L2ARC, is there a "tuneable" that I can test which instructs the ARC to prioritize metadata? I upgraded from 16GB to 32GB ECC RAM. Yet there is zero change in this behavior. This keeps happening: I run regular rsync tasks from a few local clients, which is...

www.truenas.com

But you would be setting secondarycache, not primarycache in case you don't read the whole thread until the part where it talks about L2ARC.

 

[ChrisRJ]

I have the suspicion that my SSD is failing

What makes you think that?

 

[Whattteva]

Should I really be wasting the 2x nvme on the system setup, it almost seems like all of the config and work goes into the Apps so I should really move my mirror to Apps and then just use a single nvme for System?

Even 1x NVMe is overkill really. I'm running the boot pool on an ancient 60GB SSD that is like 10+ years old. Also, as long as you backup your config file (a very small file), the boot pool is completely disposable and can be regenerated from scratch by reinstalling + uploading the config file in 5-10 minutes max.

Because if System fails I just replace and reinstall truenas and then discover both pools and then all of my data and apps, containers, vm's all come back and work just fine?

Yep, just like I said earlier above. Reinstall + restore config file.

 

[bigjohns97]

What makes you think that?

Previously used this drive with a debian VM on a Hyper-V instance and was getting disk corruption complaints from elasticsearch whenever it would rotate indexes.

Also in the last couple of days since I installed TrueNAS and starting deploying apps and VM's on this mirror with it and the nvme drive it has complained about being degraded.

Now this could be the fact that this SATA SSD is on the LSI controller with all of the other mechanical drives and the NVMe is on the motherboard but honestly this EVO drive is just old and busted and needs to be put out to pasture.

 

[bigjohns97]

Even 1x NVMe is overkill really. I'm running the boot pool on an ancient 60GB SSD that is like 10+ years old. Also, as long as you backup your config file (a very small file), the boot pool is completely disposable and can be regenerated from scratch by reinstalling + uploading the config file in 5-10 minutes max.


Yep, just like I said earlier above. Reinstall + restore config file.

Just did this and what an awesome experience you don't even have to rediscover the pools or mess with any config, just boots right back up like nothing happened.

Really impressive.

 

[bigjohns97]

BTW I was dealing with quite a few issues here.

RAM was running it's XMP or whatever the AMD equivalent is and that seemed to be causing some issues as well as the SATA SSD being bad.

So ended up just going with a single boot NVMe and a NVMe mirror for apps with the 6x RAID10 array with a hot spare.

I have some ECC memory arriving soon, hopefully that will be the last time I have to touch this thing for a while and I can actually enjoy the fruits of my labors.

Thanks to all of the assistance everyone!

 

[Whattteva]

BTW I was dealing with quite a few issues here.
RAM was running it's XMP or whatever the AMD equivalent is and that seemed to be causing some issues as well as the SATA SSD being bad.
So ended up just going with a single boot NVMe and a NVMe mirror for apps with the 6x RAID10 array with a hot spare.
I have some ECC memory arriving soon, hopefully that will be the last time I have to touch this thing for a while and I can actually enjoy the fruits of my labors.
Thanks to all of the assistance everyone!

Glad it worked out for you. Have fun enjoying the fruits of your labor!

 

[bigjohns97]

Actually spoke too soon. Just checked and the same thing happened, failures on both pools with the same counts on each disks but zero counts on the devs themselves.

I think I am dealing with lack of ECC issues. I'll know for sure in a couple of days when the new memory gets here.

 

[ChrisRJ]

I think I am dealing with lack of ECC issues. I'll know for sure in a couple of days when the new memory gets here.

While I can not rule this out, it is more likely something with the PSU, the motherboard, or another component.

 

[bigjohns97]

While I can not rule this out, it is more likely something with the PSU, the motherboard, or another component.

I really hope not.

What makes you say this?

 

[ChrisRJ]

What makes you say this?

Having spent a couple of decades with things like this.

 

[bigjohns97]

Having spent a couple of decades with things like this.

I guess I will try PSU after this but if that doesn't work I am moving to Intel which means this mobo/cpu/ram combo all becomes useless.

 

[bigjohns97]

@ChrisRJ got my ECC ram in today and it was the previous RAM that was bad, after realizing that the new sticks fixed everything went back to the old ones and ran memtest on them and it failed within seconds.

Surprised anything worked on those old sticks.

 

[ChrisRJ]

Glad it works for you now

It would be great, if you could mark this thread as solved.