Keep single raidz1 or switch to two mirrors?

[blacksteel75]

Hi all,

Looking for recommendations on whether to change my existing 4 disk raidz1 into two mirrors, or leave as is. I'm interested in pros/cons of changing the pool configuration, or if this is even worth the effort? Here's my setup:

TrueNAS SCALE (latest)
- HPE Microserver Gen10, 64GB RAM
- (1) 500Gb NVM (pool used for ix-applications and VMs only)
- (1) raidz1 pool made of (2) 2TB, and (2) 4TB Seagate IronWolf drives.
-- The reason for the two larger 4TB drives is because I had to replace two of the original 2TB drives a year ago. So I am not using the full storage of those 4TB disks because raidz1 will only use the size of the smaller disks. Works fine, but can't access all the space.
-- Currently using only about ~3.2TB of the raidz1 pool
- All datasets are replicated/backed up daily to an offsite server
- I don't anticipate needing more storage in the future. Much of my pool space is actually used for TimeMachine backups and I'm not adding more media.

Option 1: Keep raidz1 and expand drive sizes.
- Upgrade the (2) 2TB drives to (2) new 4TB drives, and expand the pool.
- Way more storage than I anticipate needing anytime soon, but nice to have all drives of the same size and similar ages in the same pool.
- Cost is about $160 for the (2) new 4TB disks.

Option 2: Go to mirrors with existing disks.
- Convert the (2) 4TB drives to a mirror, and the (2) 2TB drives into another mirror
- This would increase my storage space a bit, I believe beyond what I have currently in the 2TB raidz1 setup.
- No cost, but I would need to figure out how to migrate the setup.

Option 3: Do nothing for now (until something fails)
- Always an option, but I'd rather make the change now.

I've seen a few posts mentioning that it isn't good to have drives larger than 2TB in a raidz1 pool, and others saying that it doesn't really matter, so I'm looking for a "poll" from the experts on the options above.

And, if "Option 2", what's the recommended way to make this change? I'm unclear of the steps to take.

I'm not particularly interested in performance, since the pool is mostly used for backups and longer-term storage. Very few users (home lab), so not optimizing for throughput or scale.

Thank you in advance!

 

[winnielinnie]

And, if "Option 2", what's the recommended way to make this change? I'm unclear of the steps to take.

Are you implying that you want to keep your data intact as you do this conversion? Because you need to basically destroy your pool to go from 1 x RAIDZ1 to 2 x mirrors using the same drives.

The only real "steps" would be to destroy the pool, and then build a new pool with a two-way mirror of 2TB drives + a two-way mirror of 4TB drives. Then transfer (rsync, cp, zfs send, etc) the data from a backup to populate this new pool.

 

[blacksteel75]

Are you implying that you want to keep your data intact as you do this conversion? Because you need to basically destroy your pool to go from 1 x RAIDZ1 to 2 x mirrors using the same drives.

The only real "steps" would be to destroy the pool, and then build a new pool with a two-way mirror of 2TB drives + a two-way mirror of 4TB drives. Then transfer (rsync, cp, zfs send, etc) the data from a backup to populate this new pool.

yeah, if option 2, i would want to keep the data intact. i thought i saw someone mention removing 1 drive from the pool so it runs in degraded mode. then format the removed drive (4TB) as a new pool, then replicate from the degraded raidz1 pool to this new drive. then when that's done, destroy the raidz1 pool and attach the other 4TB as a mirror. sounds risky, but i do have daily offsite backups. if this is a doable process?

 

[winnielinnie]

It's "doable", but I personally wouldn't do it. As long as you're comfortable with the risk, and you are 100% certain that your backup is not only up-to-date, but also accessible and restorable, then go for it?

Keep in mind that during multiple moments, including a long-lived moment where there is much I/O involved (replication), you'll be dancing on non-redundant vdevs. (I.e,, a degraded RAIDZ1 and a single-drive stripe.)

EDIT: Also forgot to mention, your data will not be balanced across vdevs after you're done. (It will be your 4TB mirror vdev that's holding all your data. And when you add your 2TB mirror vdev, it will essentially be holding nothing. Thanos will not be pleased.)

If you want the best balance, you'd have to build your 2 x mirrors, and then copy the data over from your backup.

 

[sfatula]

You could remove drive as you said, I would recommend in addition to making sure your backup is up to date, is to scrub the pool before doing so for the best chance of success.

 

[blacksteel75]

Thanks. I'm leaning towards Option 1 and just putting the other two 4TB drives into my raidz1 pool, just for simplicity sake. I'd go through all this for Option 2 if it was significantly better.

 

[sfatula]

Option 2 will provide more speed with multiple vdevs and outperform raidz1 for the most part and iot's cheaper, so there is that. Easier to expand in the future. Slightly more fault tolerance and much faster resilvers. There's your advantages of option 2 but it's all up to you.

 

[blacksteel75]

gotcha. thanks for the insights.

for the things I've seen about using disks larger than 2TB in a raidz1 pool, is there any way to understand how risky that is? i.e. how much more risky than my current raidz1 pool if I increased all disks to 4TB?

 

[winnielinnie]

The idea is that the larger the member disks, the longer it will take to resilver, and hence there's a wider window open for another drive to fail during that precarious time when you have no redundancy.

(It really applies to how much data you have stored, since a resilver of a mostly empty pool won't take long.)

 

[sfatula]

for the things I've seen about using disks larger than 2TB in a raidz1 pool, is there any way to understand how risky that is? i.e. how much more risky than my current raidz1 pool if I increased all disks to 4TB?

There are two camps here on these forums regarding Raidz1. One camp states that any Raidz1 is bad and they worry about having a single URE during rebuild, as the drives are worked harder than ever during the resilver. Some cite the old article about Raid 5 is dead, which is a Raid article from many many years ago. Which of course only partially applies to ZFS as it's not the same as old style Raid.

The other camp says risk is lower than camp 1 says as if you have enterprise drives which support things like tler, then, the risk is actually nothing like what the other camp says as any URE would simply affect a file and zfs would even tell you what file and you could restore it. Since it's not Raid (it's ZFS), a URE does not stop a rebuild. And assuming you do weekly scrubs which verify the data anyway, the risk would seem much lower than is stated.

For me, I am more in camp 2 than 1. However, I do have multiple backups and understand the risk. It's situational though. In some cases, I'd be in camp 1.

I've literally been using Raid 5 (much less fault tolerant than ZFS raidz1) since about when mdraid came out on Linux (a long time in other words), never lost a single array, probably had dozens are arrays and machines (there wasn't even a raid 6 back then for the most part). We used software and hardware based Raid cards. But people lose arrays/pools all the time, this is not to imply this is untrue of course. Just my experience. But I did do weekly scrubs as well. Maybe I was an outlier, very lucky. Of course, those were smaller drives too as large drives didn't really exist.

So, the choice is up to you in the end. For me as a home user, I am not as much concerned about losing a file or a pool. I'd live, minor inconvenience. However, if I was working at a business, I would not even consider it. I seriously doubt I'll lose a pool though. I was using Raidz1 with 4TB drives on ubuntu for many years, lost many a drive (I'd guess 6 over the years) and in my case, they were all older drives. I had one pool that was 3 vdevs of 3 4TB dtives with a spare. But never lost a file or pool. However, I kept vdevs at 3 drives which I feel is sort of a sweet spot. 4 probably isn't much worse. But again, I had 3 or 4 different backups (some onsite some off) and I did weekly scrubs and I did monitor smart data and reports of drive errors and did something about it. I don't lose sleep about those dreaded URE's in my specific case. For what that is worth!

I've also segregated my data into different priorities (different datasets) so the most important data has the most backups and types of backups. The least important data like temp workspace, transcoding directories, etc are never backed up as couldn't care less if those get lost in a failure. It was surprising to me when I actually looked at all the data I was storing, how much was in the don't care camp. But every use case is different. For now, am using multiple vdevs of mirrors for speed.

 

[blacksteel75]

Thanks all, this is really helpful info. Much appreciated!