Remove vdev from Pool without destroying pool

[wizengamot]

I have a Pool made up of sets vdevs, each vdev is a RaidZ1 of 4 disks. I ran into some problems awhile back and had to replace a few disks that FreeNAS was telling were failing. I replaced the disk in a given vdev without removing the old disk immediately. Now the disks that I replaced show to good again and I don't know which ones FreeNAS wanted me to remove. That is the problem with the vdev that shows 7 disks in the attached screenshot.

The second problem is that I have a single vdev made up of 4 2tb hard drives that are now failing, but the failures are showing up in the log as shown in the second image but they don't seem to appear in the volume. The da3 the refers to appears to be the raidz1-2 vdev disk da3p2 which is a member of the previously mentioned 2tb array.

What I would like to do, in the absence of money to solve the problem by replacing the drives would be to remove that vdev completely in a manner that allows me to retain the volume without affecting the data. My available disk space would drop by 6tb or so which will not be a problem, as the system list 13.1TB available at the second volume level and 20.1 tb available at the top level.

For the first problem, I would like to find a way to identify which disks should be removed from that vdev so that it comes back down to 4 disks.

If anyone can help me that would be great. Even pointing me to documentation that so far has eluded my searches (yes I have looked, just not successfully) and it likely has a lot to do with the fact that I am not sure what I am looking for.

Thanks
B

 

[garm]

What I would like to do [..] would be to remove that vdev completely in a manner that allows me to retain the volume

Unfortunately you can’t do that.

For the first problem, I would like to find a way to identify which disks should be removed from that vdev so that it comes back down to 4 disks.

Well, to start you could post (in code tags [CODE] like this [/CODE] ) the output of zpool status so we can see what you see.

 

[wizengamot]

I set the command window as tall as it would go and I was able to copy the following. I'll have to get putty for a more complete listing. I'll edit this post shortly with the output from putty.

Code:

Shell
action: Restore the file in question if possible.  Otherwise restore the                                                           
        entire pool from backup.                                                                                                   
   see: http://illumos.org/msg/ZFS-8000-8A                                                                                         
  scan: resilvered 14.1T in 0 days 14:49:08 with 25 errors on Tue Aug 20 13:21:27 2019                                             
config:                                                                                                                           
                                                                                                                                  
        NAME                                              STATE     READ WRITE CKSUM                                               
        Vol01                                             ONLINE       0     0    25                                               
          raidz1-0                                        ONLINE       0     0     0                                               
            gptid/34acad66-3e9b-11e8-b1f4-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/a8216798-383d-11e8-bfb2-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/a957c119-383d-11e8-bfb2-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/aa80761e-383d-11e8-bfb2-1c6f6596a2cb    ONLINE       0     0     0                                               
          raidz1-1                                        ONLINE       0     0    50                                               
            replacing-0                                   ONLINE       0     0     0                                               
              gptid/31e848b8-3f8f-11e8-b1f4-1c6f6596a2cb  ONLINE       0     0     0                                               
              gptid/8bb51e4c-b407-11e9-8390-1c6f6596a2cb  ONLINE       0     0     0                                               
            replacing-1                                   ONLINE       0     0     0                                               
              gptid/32ae3d70-3f8f-11e8-b1f4-1c6f6596a2cb  ONLINE       0     0     0                                               
              gptid/14af1a1c-ac8f-11e9-9068-1c6f6596a2cb  ONLINE       0     0     0                                               
            replacing-2                                   ONLINE       0     0     0                                               
              gptid/3374e0eb-3f8f-11e8-b1f4-1c6f6596a2cb  ONLINE       0     0     0                                               
              gptid/9ff1ea58-abea-11e9-9068-1c6f6596a2cb  ONLINE       0     0     0                                               
            gptid/7af8b9b7-ab4d-11e9-9068-1c6f6596a2cb    ONLINE       0     0     0                                               
          raidz1-2                                        ONLINE       0     0     0                                               
            gptid/fad8d474-6ad8-11e8-af15-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/fef53299-6ad8-11e8-af15-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/00d86405-6ad9-11e8-af15-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/037d8751-6ad9-11e8-af15-1c6f6596a2cb    ONLINE       0     0     0                                               
          raidz1-3                                        ONLINE       0     0     0                                               
            gptid/65ae46be-00c4-11e9-a342-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/67c480ff-00c4-11e9-a342-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/6872114f-00c4-11e9-a342-1c6f6596a2cb    ONLINE       0     0     0                                               
            gptid/69e523e4-00c4-11e9-a342-1c6f6596a2cb    ONLINE       0     0     0                                               
                                                                                                                                  
errors: 1 data errors, use '-v' for a list                                                                                         
                                                                                                                                  
  pool: freenas-boot                                                                                                               
state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0 days 00:04:00 with 0 errors on Wed Aug 21 03:49:00 2019                                             
config:                                                                                                                           
                                                                                                                                  
        NAME        STATE     READ WRITE CKSUM                                                                                     
        freenas-boot  ONLINE       0     0     0                                                                                   
          mirror-0  ONLINE       0     0     0                                                                                     
            da11p2  ONLINE       0     0     0                                                                                     
            da12p2  ONLINE       0     0     0                                                                                     
                                                                                                                                  
errors: No known data errors 

 

[garm]

Okey, you have pool checksum errors. Those won’t go away and this is a good example why raidz1 simply isn’t enough. You need to back up your data and rebuild the pool. If you intend to reuse the disks you should redo the burn in procedure.

 

[wizengamot]

I thank you for your reply. Can I ask what may be a stupid question. I have heard much about the ZFS being the wonderful new filesystem that is way way better than BTRFS such as what is used by Synology. However, based on my use of ZFS to date it appears that BTRFS is in many ways better. I have seen in posts that an 8 drive array is the largest ZFS array to use. With certain Synology systems I can use up to 36 drives with a single volume. If I get a bad drive with BTRFS I find that the system will tell me exactly which drive to remove and replace. With ZFS its a little more complicated, I grant not much but for the newbie anything that is more complicated can be a chasm as large as the grand canyon. Based on what I have been told above it is NOT possible to shrink an array of ZFS vDevs, but shrinking an array of disks based on BTRFS is possible (at the least it is possible on Ubuntu).

I don't want the question to sound impertinent because that is not my goal, I truly would like an expert on ZFS to weigh in and let me know what exactly is so great about ZFS vs BTRFS. I think that knowing what ZFS experts think about ZFS as compared to BTRFS would help me make the decision to buy (or not) a 36 drive Synology system.

Please do not take my question as anything other than an honest request for information as I have no intention here other than to learn. A google search looking for comparison data on ZFS vs BTRFS did not really turn up anything.

 

[garm]

ZFS is a more mature system and isn’t crippled by serious design flaws https://btrfs.wiki.kernel.org/index.php/RAID56

To date there has been no serious bugs in ZFS as far as I’m aware.

If you want to know more about the underlying architecture there are a few YouTube videos on conference presentations from the devs detailing interesting aspects of the code. Or you can just read the code.

ZFS is an enterprise solution. It was born with a specific set of requirements. Stability and data integrity was way up there on the list.

Shrinking a pool would mean that already written data would be rewritten to the pool and that was not a process that was originally considered. There is work ongoing for some manipulation in regards to extending a raidz virtual device and removing virtual devices but it hasn’t made it to the stable code yet.

ZFS max pool size is 256 trillion yobibytes.
How many disks you should put in a single vdev is however limited to the performance of the disks. The way vdevs are structured means that the iops performance of a vdev is roughly the same as for the slowest disk in the set. Originally the vdevs where mirrors, raidz came later and are limited in comparison.
So if you put 32 drives in a single raidz vdev you will have the performance of a single drive with the size of 32 drives. This will make your life miserable in terms of doing basically anything. But system design is still up to you and if you do the math you might get away with it depending on your usecase and hardware and configuration.

Synology isn’t using BTRFS for raid, instead they use mdadm array to store BTRFS on. It’s not without its limitations https://en.wikipedia.org/wiki/Btrfs

But everything is a cost balance exercise, it’s up to you to decide the value of your data and the risk you expose it to. If disk removal is a critical feature for you then ZFS won’t be enough at the moment. If data integrity is paramount there is nothing that beats ZFS.

Personally, the way I handle my system it’s inconceivable to risk non backed up data by removing disks. And if you do back up data you can still reorganize a ZFS pool and then read back from backup. So I really don’t see the big fuzz about agile disk management in a NAS..

 

[danb35]

I have heard much about the ZFS being the wonderful new filesystem

ZFS was introduced in 2005, so it's hardly "new". btrfs isn't that much younger, but ZFS has seen much more in the way of development and testing.

As @garm says, Synology isn't using btrfs for its volume manager; it's using the btrfs filesystem on top of the LVM volume manager. This is probably a good thing, as it avoids many of the big problems with btrfs (things like being coded with a write hole for parity RAID, parity not being checksummed, and parity RAID in general being terribly fragile). The problems with btrfs are severe, pervasive, and long-stanging enough that RedHat, after dumping tons of money into the project, has pulled their support.

extending a raidz virtual device and removing virtual devices but it hasn’t made it to the stable code yet.

Extending RAIDZ isn't there yet, but vdev removal is (it's in FreeNAS 11.2, though not in the GUI)--but only for pools where all vdevs are either single disks or mirrors. From what I understand of how they did that, it's highly unlikely it will be extended to pool where there are any RAIDZ vdevs.

 

[wizengamot]

I would like to thank you all for your responses to my inquiry. From what I understand from what was written is that ZFS is far more resilient than other platforms. The data that I am storing on my FreeNAS is bulky and not especially valuable. I would not enjoy losing it, but it would not be life altering. As such I do not have a way to back that data up at present. I am going to re-design my vdev structure and I would be interested to know what is the best structure to prevent the loss of data.

In the current structure, raidz1 sets of 4 disks, I found that I "lost" portions of or at the very least the drive was listed as degraded on three drives in a single vdev and I did not lose any data. I know for a fact that had that been a standard RAID I would have lost it all. If I was made of money maybe I would get a 12 drive DROBO but I am not so I chose FreeNAS. Based on what @garm and @danb35 wrote above this was a good decision.

Since I cannot reduce my HDD count in the original vdevs I still have the vexing problem of the 2TB drive array. When I was building the system, I stupidly used some old desktop drives I had lying around to get a few extra cheap GB and these are the drives in that 2TB array (I call it the 2TB array but it is a raidz1 of 4 2TB disks ... 6TB?). Would it be possible to perform the following: Replace Disk 1 of this 2TB array, resilver, replaced disk 2, resilver ..etc until the disks are all replaced. Also, could I say used 4 or 8tb disks for this (my standard is WD Reds) and at the end of this have a larger more stable vdev? 4 - 4TB disks for a total of 12TB in the array instead of 6TB?

I would also be interested in finding out what is recommended in terms of vdev arrangement. My system is limited to 24 hard drives as I have two cards in the system each supporting 8 drives for a total of 16 and motherboard ports I have a total of 8 for a max of 24 drives. I am booting the system off of a RAID1 of USB sticks. In keeping the max drive limit in mind and my preference for data integrity balanced with a need for space what would you recommend for vdevs should I decide to rebuild the array? What I would say is this. If you had 24 WD 8tb Reds and were building your own FreeNAS setup from scratch, what vdev arrangement would you use for your own rig given that space and resiliency should be evenly matched.

These are some of the basic questions I came up with that might help me answer the above question in addition to others ideas.
1. Is a raidz1 with 4 drives just as resilient as a raidz2 with 8 drives?
2. Do both arrays listed in #1 give the same space?

The only performance I am after is the ability to copy two files at once from the array to another computer (two different computers) and have that copy occur at a space higher than 20MB per second on each copy. The system as it stands already does more than that but that would be "hard floor" of performance.

The very last question I have is in my configuration that I posted you can see a vdev with 7 drives. What occurred here was I saw a drive was degraded and I used the replace disk feature and allowed it to resilver each time. The disks I used were of the same capacity as the original. Now I am not sure which disks to remove. I thought that there might be a some sort of convention I was not aware of that would allow me to ID the serial numbers of the drives it was safe to remove. Of course, all the drives that once said degraded are no longer listed as degraded and I have no idea as to why that is.

Thanks again for all the assistance, you guys have been great and special thanks to both @garm and @danb35 for their answers. They were informative.

 

[N20Visuals]

since this thread is a couple of years old now, i just wanna ask if removing a top-level mirror vdev is possibe in truenas scale? the underlying zfs should have the same functionality i assume?

at the moment i have a configuration of 4 Mirror-Vdev's, striped, as recommended for easiest expandability.
now i got my hands on some 12tb drives and have lots of storage to spare and due to rising energy costs i'm trying to save power everywhere i can.

my pool constists of:
2x 12tb
2x 12tb
2x 4tb
2x 4tb

is it possible (if enough free storage space is available of course) to remove one or even both of the 4tb mirrors without destroying the pool?

 

[danb35]

i just wanna ask if removing a top-level mirror vdev is possibe in truenas scale?

It's possible in FreeNAS 11.3 11.2 and up, TrueNAS CORE 12.0 and up, and all versions of SCALE. It wasn't implemented in the GUI in 11.3, but it is in CORE 12 and up and in SCALE. IIRC it's in the pool status page.

 

[Patrick M. Hausen]

According to the documented state of affairs it should be possible in SCALE and in CORE.

Code:

root@freenas[~]# truncate -s 1g /mnt/hdd/disk0
root@freenas[~]# truncate -s 1g /mnt/hdd/disk1
root@freenas[~]# truncate -s 1g /mnt/hdd/disk2
root@freenas[~]# truncate -s 1g /mnt/hdd/disk3
root@freenas[~]# zpool create testpool mirror /mnt/hdd/disk0 /mnt/hdd/disk1 mirror /mnt/hdd/disk2 /mnt/hdd/disk3
root@freenas[~]# zpool status testpool
  pool: testpool
 state: ONLINE
config:

    NAME                STATE     READ WRITE CKSUM
    testpool            ONLINE       0     0     0
      mirror-0          ONLINE       0     0     0
        /mnt/hdd/disk0  ONLINE       0     0     0
        /mnt/hdd/disk1  ONLINE       0     0     0
      mirror-1          ONLINE       0     0     0
        /mnt/hdd/disk2  ONLINE       0     0     0
        /mnt/hdd/disk3  ONLINE       0     0     0

errors: No known data errors
root@freenas[~]# zpool remove testpool mirror-1
root@freenas[~]# zpool status testpool
  pool: testpool
 state: ONLINE
remove: Removal of vdev 1 copied 50K in 0h0m, completed on Thu Oct 20 13:16:01 2022
    192 memory used for removed device mappings
config:

    NAME                STATE     READ WRITE CKSUM
    testpool            ONLINE       0     0     0
      mirror-0          ONLINE       0     0     0
        /mnt/hdd/disk0  ONLINE       0     0     0
        /mnt/hdd/disk1  ONLINE       0     0     0

errors: No known data errors

HTH,
Patrick

 

[HoneyBadger]

top-level mirror vdev

Just to expand on the already existing answers (and the very elegant file-based demonstration by Patrick) - vdev removal is only possible if all top-level vdevs are mirrors (or stripes). Having any RAIDZ vdevs in the pool, even if they aren't one you're trying to remove, makes vdev removal no longer possible.