Need Help with ZFS Pool! Expanding Mirror VDEV with a Single Drive

[college-jolt0a]

Hey everyone,

I'm currently facing a bit of a conundrum with my ZFS pool setup, and I'm hoping that some of you knowledgeable folks out there can lend a hand. I have a ZFS pool with a mirror VDEV configuration (2 x 4TB), and I'm looking to expand it by adding a single drive (1 x 6TB). Here are the details:

Current Setup:

Mirror VDEV: Drive A (4TB) + Drive B (4TB)
Total Usable Capacity: 3.64TiB (mirrored)
Desired Expansion:

Add a new Drive C (1TB) to the existing pool
Reconfigure the mirror VDEV to include Drive C
Increase the usable capacity of the pool by configuring it to RAID1?
I understand that ZFS allows for expanding a pool by adding VDEVs, but since I have only one other drive, I need clarification.
I've done some research, but I couldn't find a definitive answer or a step-by-step guide for this specific scenario.

If any of you have experience with expanding ZFS pools in a similar way, I would greatly appreciate your guidance. Here are a few questions that I have:

Is it possible to expand a mirror VDEV with a single drive? I do see extend button in the UI. What does it do?
What is the recommended procedure for adding a single drive to an existing mirror VDEV?
Will the data from Drive A and Drive B be automatically rebalanced across the new Drive C?
Are there any precautions or best practices that I should be aware of before attempting this expansion?
I'm open to any suggestions, tips, or resources that you can provide. Your expertise will go a long way in helping me achieve a successful expansion of my ZFS pool. Thank you all in advance for your support!

My current pool is 80% full, and I want to keep the data, and I don't have any additional drive lying around, So deleting and recreating the pool would be very difficult.

 

[Arwen]

No, you can not expand a 2 disk Mirror vDev / pool by adding one disk.

Their is a complicated and error prone way to convert your 2 disk Mirror into a 3 disk RAID-Z1. But, in my opinion, if you have to ask details, you are likely to make a mistake resulting in data loss. ZFS is not that flexible and was not designed for the really low end. It CAN work at the low end, (low end meaning just a few drives), and work well. It is just that ZFS was designed for data center servers.

Last, except for future proofing, using a single 6TB drive does not help. You would not be able to use the extra 2TB of space.

 

[college-jolt0a]

That's a bummer and sad. What should I do now? What will be the additional storage I have by extending the mirror VDEV by 6TB? Will I have an additional 4TB? And what could I do in future if I buy another drive? What do you suggest?

 

[danb35]

No, you can not expand a 2 disk Mirror vDev / pool by adding one disk.

You could add a third disk to the mirror, resulting in a three-way mirror, but as you say that wouldn't increase the capacity of the pool. You could also add a single-disk vdev to the pool--that would increase its capacity, but at the risk of all data being lost when (not if) that single disk dies. If there's a thought toward adding a fourth disk in the (very) near future, that might be an acceptable risk, but...

 

[Arwen]

That's a bummer and sad. What should I do now? What will be the additional storage I have by extending the mirror VDEV by 6TB? Will I have an additional 4TB? And what could I do in future if I buy another drive? What do you suggest?

No, as I said, you can not extend the Mirror vDev by 6TB and get 4TB or 6TB of more space. A Mirror vDev in ZFS is more or less immutable / un-changable sizewize. You can change the amount of redundancy in a Mirror vDev by adding or removing Mirror elements. But that does not increase storage, just redundancy.

If you do buy another drive, preferably 6TB, you can add a second Mirror vDev of 6TB. This assumes you have the following:

• SATA port for a second 6TB drive
• Enough power from your power supply for a second 6TB drive, and power cable for the drive
• Mounting location for a second 6TB drive

This can be done without harming your existing data on the 2 x 4TB disks. You can also take the 2 new 6TB disk and make a separate ZFS Mirror pool. Depends on your needs.

Next, if you don't have the extra SATA port, power or mounting slot for a second 6TB drive, you can take the 2 x 6TB drives and replace the 2 x 4TB drives, one at a time. After the replacing the last drive in the Mirror vDev, storage would jump to 6TB total, giving you 2TB more space. BUT, you have to replace ALL elements in a Mirror vDev, (just 2 x 4TB drives in your case), for this to work. Again, no harm to your existing data if done right.


What you are experiencing is an example that others have faced, lack of flexibility in ZFS for the low end of servers. (It also affects higher end servers too, but sometimes money or planning works around the problems.)

This is partly where ZFS gets a poor reputation. Not because of data loss, but lack of flexibility.

ZFS is also sometimes blamed for data loss, when in reality, that same data loss would have occurred on every other file system if given the same setup.

 

[college-jolt0a]

Thanks for the clear explanations. Since I only have a budget build, it makes sense for me to switch to Unraid, which will be more storage efficient.

 

[winnielinnie]

Since I only have a budget build, it makes sense for me to switch to Unraid, which will be more storage efficient.

I don't understand this.

Are you implying to go the route of essentially mirror + stripe? (Or JBOD / stripe elsewhere?)

You can do this in ZFS as well, but it's dangerous because the "one additional drive" has no redundancy. You lose it, you lose everything. Your pool would be (precariously) comprised of two vdevs. (i.e, 2-drive mirror + 1-drive stripe). Your pool's overall integrity hinges on that 1-drive stripe. (But you will gain an extra 6TB of usable storage for the pool.) I think the TrueNAS GUI won't let you do this because of how dangerous it is to your data.

@danb35 explained this above. So I'm still confused how Unraid is "more storage efficient"?

Colorful TDLR:
Green good
Red bad
Good + Bad = Bad


If the idea is rather to create a RAID-Z1 vdev (from 3 drives), you'd still have to start all over again if you went with Unraid to configure something similar.

 

[college-jolt0a]

As you mentioned, I can't go with mirror + stripe (it's dangerous). I will have to buy another drive, and if I buy a new one, I will have 2 VDEVS with 4TB and 6TB capacity giving me a total of 10TB.

Based on my research, Unriad uses the largest single drive as parity, eventually giving me more space with four drives. Isn't it? Correct me if I am wrong.

 

[winnielinnie]

It sounds like you're referring to how ZFS does not yet provide "RAIDZ expansion", while Unraid already does allow something similar in scope?

If that's the case, "mirror" vdevs are unrelated. You'd build a RAIDZ (-1, -2, -3) vdev from the start, and continue to expand its capacity, one drive at a time as needed. (This, however, is not available with ZFS yet. Who knows when it may ever arrive?)


EDIT:
I'm not familiar with how Unraid achieves this, and I'd be skittish if it's some sort of uncommon and/or proprietary method.


EDIT 2:
So Unraid uses some weird frankenstein combination of proprietary and opensource, in which you assemble a JBOD (which can include varying drive capacities), while you tag one or more "large" drives to hold the "parity data". (Data is not "striped across" all drives, as is the case with traditional RAID or ZFS vdevs. You tell Unraid to use a specific drive(s) to serve only to hold the parity data.)

I honestly would not use it to store valuable, irreplaceable data for the longterm. It really does come off as "I have an assortment of different drives I don't want to throw away, but I need to use them to store a bunch of data that might not be that important to me." I'd trust Synology more than Unraid.

 

[Arwen]

Thanks for the clear explanations. Since I only have a budget build, it makes sense for me to switch to Unraid, which will be more storage efficient.

Probably.

As I have said in the past, their is no NAS software to rule them all. Each of the major players have pros and cons.

 

[2twisty]

I am a former UNRAID user, so let me add to this:

Unraid creates a RAID-5/6 ish array of disks. As mentioned above, the data is not truly striped across the drives the same way that ZFS and RAID do. It does this at the file level, not the block level. So you won't have a single file spread across two disks (except for the parity stuff on the parity drive).

The system tries to balance the data across all the drives, but they can get out of balance, and you have to rebalance them if you want the data spread evenly.UnRAID's cache drive is NOTHING like an SLOG. It's actually a true write cache. Data comes in and is written to the cache. Periodically, a cron job runs "mover" which moves the data from the cache to the disk array.

The real "magic sauce" of UNRAID is that it presents all this to you as a single filesystem. It handles where the data goes in the background.

Unlike ZFS and RAID that work at the block level, you can actually remove a disk from the array and read it in a regular Linux box. This means that a disk loss means that at worst, you lose only the data on that disk. Your parity disk can rebuild the missing data.

UnRAID is not a bad NAS software if you don't need a ton of parallel access to the same data. This is because any given read/write operation will ONLY occur on one disk.

I switched to TrueNAS because I wanted the added data protection that ZFS gives in the form of bitrot detection. ZFS is much better at finding errors in your data and quietly fixing it. You don't get this level of protection from UNRAID or standard RAID. I also wanted something that would be a bit faster with the data because I wanted to use it as a SAN at the same time for my VM storage. UnRAID is not really suited for that.

Given your budget and hardware that you've mentioned, I'd probably suggest UnRAID. It's not free, so there is the cost to purchase it. For people who want "one step up" from your typical consumer NAS to hold your personal files, UnRAID's not a terrible choice.


PLEASE NOTE: I left UnRAID a few years ago, so there may be improvements that address some of my reasons for leaving. Most notably, UnRAID now supports ZFS. However, in your situation, I would not switch to UnRAID and then try to use ZFS, because you'd be in the same situation you are now; the limits talked about above are limits of ZFS, not TrueNAS.