troubleshooting mirror vdev won't expand

[Saoshen]

I appear to have a similar situation as @ https://www.truenas.com/community/t...th-larger-drives-but-cant-expand-pool.112702/ but that thread ended with inconclusively.

I have a pool, tank1. It is 6 disks, 3x 2-wide mirrors.

originally, it was 6x 8tb drives.

I replaced the first mirror vdev with 2x 10tb drives, and it expanded as expected.

I replaced the second mirror vdev with 2x 10tb drives, and it has NOT expanded as expected.

Code:

 sudo zpool list -v tank1                                                                                                                                                                                                                                                        sauron2: Sun Nov  5 09:21:21 2023

NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank1                                     23.6T  21.6T  2.01T        -         -     5%    91%  1.00x    ONLINE  /mnt
  mirror-0                                9.08T  7.56T  1.52T        -         -     5%  83.3%      -    ONLINE
    4aa5ef46-b636-4331-80bd-f99e2f965ebc  9.09T      -      -        -         -      -      -      -    ONLINE
    af41c1c6-f77c-40f4-aff1-c1455c21d549  9.09T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                7.27T  7.18T  84.5G        -         -     9%  98.9%      -    ONLINE
    f5f746be-5bb4-4e37-8f28-07d1313a4118  7.28T      -      -        -         -      -      -      -    ONLINE
    f119d4ae-0783-4a22-9933-cbafd18b5432  7.28T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                7.27T  6.85T   426G        -         -     3%  94.3%      -    ONLINE
    39d8df03-b26c-4053-a28a-78e53c8be433  7.28T      -      -        -         -      -      -      -    ONLINE
    3dd8f8ec-2e88-4f77-a380-586fdb60265a  7.28T      -      -        -         -      -      -      -    ONLINE

Code:

sudo zpool status tank1                                                                                                                                                                                                                                                         sauron2: Sun Nov  5 09:21:56 2023

  pool: tank1
 state: ONLINE
  scan: scrub repaired 0B in 18:04:03 with 0 errors on Sun Nov  5 04:47:09 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank1                                     ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            4aa5ef46-b636-4331-80bd-f99e2f965ebc  ONLINE       0     0     0
            af41c1c6-f77c-40f4-aff1-c1455c21d549  ONLINE       0     0     0
          mirror-1                                ONLINE       0     0     0
            f5f746be-5bb4-4e37-8f28-07d1313a4118  ONLINE       0     0     0
            f119d4ae-0783-4a22-9933-cbafd18b5432  ONLINE       0     0     0
          mirror-2                                ONLINE       0     0     0
            39d8df03-b26c-4053-a28a-78e53c8be433  ONLINE       0     0     0
            3dd8f8ec-2e88-4f77-a380-586fdb60265a  ONLINE       0     0     0

errors: No known data errors

Since the first vdev expanded automatically, I expected the rest to as well, but I have also tried expanding via the GUI, and via the shell.

Both completed with no error shown, however space has never expanded.

I downloaded my debug logs and went through them, but did not see anything obvious errored in any of the logs. I did see the log where it showed pool expansion commands.

Pool was originally created on bluefin. It was zfs feature upgraded some days later after the cobia upgrade, and before the first vdev expansion.

Where/what should I be looking at next for troubleshooting?

TrueNAS-SCALE-23.10.0.1

hardware:
hp proliant gl360 gen 10, 96gig ecc, 2x xeon gold 5220r 48c 96t, 8 bay sff/2.5" mixed ssd's z2.
lsi sas 2116 it mode -- 0 SAS2116_1(B1) SAS9201-16e
netapp 4243 disk shelf. 24 disks, 6 pools, mostly mirror 4 or 6 wide sets, a couple single disk vdevs as interim storage.

I also had this problem on a previous pool, I ended up moving the data off and re-creating the pool, and moving the data on, I really really don't want to waste another week or more doing the same.

Thanks for any thoughts or recommendations.

 

[Patrick M. Hausen]

Output of gpart show would be helpful.

 

[Saoshen]

hmm

Code:

sudo gpart show
sudo: gpart: command not found

no gpart or gparted

 

[Patrick M. Hausen]

Oops, sorry. SCALE. No idea ...

 

[winnielinnie]

Might work on SCALE:

Code:

 parted -l

Or use this for more info with an easier-to-read format:

Code:

lsblk -o name,partuuid,fstype,size

 

[Saoshen]

ok thanks:

the disks in question, order of pool gui list:

sdg, sdp is the affected pair.

Code:

sdf                                                                  9.1T
├─sdf1      671a37c5-1a78-437c-b52c-63eb76b98af8                       2G
└─sdf2      4aa5ef46-b636-4331-80bd-f99e2f965ebc zfs_member          9.1T

sdh                                                                  9.1T
├─sdh1      9bac6367-d172-4331-81c1-eb7800e166af                       2G
└─sdh2      af41c1c6-f77c-40f4-aff1-c1455c21d549 zfs_member          9.1T

sdg                                                                  9.1T
├─sdg1      f119d4ae-0783-4a22-9933-cbafd18b5432 zfs_member          7.3T
└─sdg2      e09f9fe4-12f1-4625-bb5c-56ab74090700                       2G

sdp                                                                  9.1T
├─sdp1      f5f746be-5bb4-4e37-8f28-07d1313a4118 zfs_member          7.3T
└─sdp2      4c2b0bc1-3be6-484a-9ef9-60855856d53b                       2G

sdr                                                                  7.3T
├─sdr1      bccec547-9fa2-4d95-90ce-e7ecd5d29386                       2G
└─sdr2      39d8df03-b26c-4053-a28a-78e53c8be433 zfs_member          7.3T

sdv                                                                  7.3T
├─sdv1      adc30e37-62b3-4564-bb4a-36cbd9d756dc                       2G
└─sdv2      3dd8f8ec-2e88-4f77-a380-586fdb60265a zfs_member          7.3T

Code:

Model: WDC WD100EMAZ-00WJSM (scsi)
Disk /dev/sdf: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2149MB  2147MB                     swap
 2      2150MB  10.0TB  9999GB  zfs

Model: WDC WD100EMAZ-00WJSM (scsi)
Disk /dev/sdh: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2149MB  2147MB                     swap
 2      2150MB  10.0TB  9999GB  zfs

Model: WDC WD100EMAZ-00WJSM (scsi)
Disk /dev/sdg: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  7999GB  7999GB  zfs
 2      7999GB  8002GB  2147MB                     swap


Model: WDC WD100EMAZ-00WJSM (scsi)
Disk /dev/sdp: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  7999GB  7999GB  zfs
 2      7999GB  8002GB  2147MB                     swap

Model: ST8000VN 0022-2EL112 SM (scsi)
Disk /dev/sdr: 8002GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      65.5kB  2147MB  2147MB                     swap
 2      2148MB  8002GB  7999GB  zfs

Model: ST8000VN 0022-2EL112 SM (scsi)
Disk /dev/sdv: 8002GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2149MB  2147MB                     swap
 2      2150MB  8002GB  7999GB  zfs

so, my 2nd vdev, has somehow reversed order of swap/zfs ?!? how does that happen?

full disks lists attached

 

[winnielinnie]

One thing that caught my attention is this, which concerns the two drives in question:

Code:

sdg                                                                  9.1T
├─sdg1      f119d4ae-0783-4a22-9933-cbafd18b5432 zfs_member          7.3T
└─sdg2      e09f9fe4-12f1-4625-bb5c-56ab74090700                       2G

sdp                                                                  9.1T
├─sdp1      f5f746be-5bb4-4e37-8f28-07d1313a4118 zfs_member          7.3T
└─sdp2      4c2b0bc1-3be6-484a-9ef9-60855856d53b                       2G

Unlike the other drives, these two had their swap devices created as the second partition (not the first partition.)

I wonder if that throws off or confuses the GUI's "expand" tool?

I wonder if you can still use the command-line to manually (and safely) expand the pool?


EDIT: I can only speak for Core, but any time I created a pool or added a mirror vdev, the 2GB swap space was always the first partition on the drive(s).


EDIT 2: This doesn't look good...

Code:

Model: WDC WD100EMAZ-00WJSM (scsi)
Disk /dev/sdg: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  7999GB  7999GB  zfs
 2      7999GB  8002GB  2147MB                     swap

There's no room for the first partition (to grow beyond 7TB), due to the creation of the swap partition starting immediately where the zfs-member ends...

I'd be worried if the TrueNAS SCALE GUI did that.

 

[probain]

If the VDEV hasn't expanded to the bigger size yet. And the partitions are being weird. Couldn't a possible way be to replace back with the smaller drives again. Removing the strange partitions on the bigger ones. Followed by another try from scratch?

Do note that I'm not saying that this is the solution. Or that you should follow that in any way. But I'm rather sharing thoughts on what I would've tried. Especially if no-one raises immediate warning flags.

 

[winnielinnie]

how does that happen?

Did you use the command-line for the second vdev replacement? The GUI should not do that. If the GUI did that, this is a serious bug that needs to be fixed as soon as possible.

 

[winnielinnie]

Removing the strange partitions on the bigger ones. Followed by another try from scratch?

My greater concern is that this even happened at all. Which means it could happen again...

I fear this is a bug in SCALE and/or the middleware code.

 

[probain]

My greater concern is that this even happened at all. Which means it could happen again...

I fear this is a bug in SCALE and/or the middleware code.

Indeed worth finding out, for sure.

 

[Saoshen]

these disks were all added via scale GUI.
and yes, I supect this is/was a bug somewhere in the middleware, since it happened previously on tank2 (which was removed and since recreated)

some history, the original pools were created piece meal over time, as I was migrating existing disks from my previous btrfs raid6 pools, to these scale mirror sets, some were created as single disks, then mirror added, then mirror vdevs added, over time, as I did the whole data shuffle between 2 systems.

adding/removing devices from btrfs, is dead simple albeit slow. I spent a couple months removing disks 1 or 2 at a time from btrfs, to add to scale as mirrors.

all data has since been migrated fully to scale, and these last 10tb disks were the final replacements for migration from my old system to this new one.

 

[winnielinnie]

@morganL or @Kris Moore, was there any recent change to SCALE that affects the swap partition creation? This seems like a serious issue if the 2GB swap is being slapped on as the second partition, essentially preventing the pool from safely expanding* its capacity with larger replacement drives.

* In regards to replacing drives in a mirror vdev.

 

[Davvo]

@Saoshen on CORE you can manually expand a pool in the WebUI, I guess you can do the same in SCALE.

 

[winnielinnie]

I guess you can do the same in SCALE.

The problem is that SCALE (using the GUI) to replace drives in a mirror vdev incorrectly put the swap as the second partition. The first partition (a zfs-member) is now limited to 7TB, and thus the vdev can only provide 7TB.

Ideally, the 2GB swap partition should be the first partition, and the remainder of the drive should use up the entirety of the remaining space (~9TB). This is always how it was handled by the GUI. Not sure if there was a recent change in SCALE's code that introduced this bug.

 

[danb35]

The problem is that SCALE (using the GUI) to replace drives in a mirror vdev incorrectly put the swap as the second partition.

No, the problem is that the first partition was created with an incorrect size. The two things may be related, but there's no inherent reason that the partition number of the swap partition would affect the pool expansion.

 

[winnielinnie]

No, the problem is that the first partition was created with an incorrect size.

I think it might be interrelated, since there's probably logic in the code that says "Create first partition with X large swap, then create next partition with the remainder."

Since (whatever bug?) created the zfs-member partition first, it probably fell back to "Just create this partition to be the same size as the other still-existing partition in this mirror, then slap the swap after that."

But it's just speculation. I wouldn't even know where to look in the source code. That's why I'm wondering if there was a recent change in the code for Cobia that affected swap creation / logic.

 

[morganL]

@morganL or @Kris Moore, was there any recent change to SCALE that affects the swap partition creation? This seems like a serious issue if the 2GB swap is being slapped on as the second partition, essentially preventing the pool from safely expanding* its capacity with larger replacement drives.

* In regards to replacing drives in a mirror vdev.

@Saoshen Suggest reporting it as a bug..... and sharing the NAS ticket ID.

 

[Saoshen]

fyi @ https://ixsystems.atlassian.net/browse/NAS-125078

Thanks for everyone's efforts.

 

[Saoshen]

hi, so it looks like there was a problem, and a fix was committed to (from what I can tell) prevent the issue, but no comment on how to fix existing affected vdevs, and the ticket closed.

will this magically fix the broken vdevs with or without user intervention?
will this require removing/readding/replacing the existing disks into the affected vdev?
will this require removing the affected vdev entirely?
will this require re-creating the whole pool?

Appreciate the work done, and any further insight into next steps (once fix is actually released).