Possible to reattach a disconnected drive to an empty pool?

[samarium]

I think the disk is enough when combined with the partition tables, sgdisk -p, of similar disks so that can be used a guide to rebuilding a partition table if you want to go that way. Even if you don't wantt to rebuild right now, it is worth doing a little probing to see if you can see if zdb might be able to see a label, if there is no label then there is very little hope.

 

[RockBomber]

Hi! Unfortunately I faced the similar problem.
What I've done (all through the web ui):
- I had zfs mirror with two 2Tb drives;
- I bought two 4Tb drives, appended one to existing mirror and waited for resilvering, it completed successfully;
- I deattached two 2Tb drives from pool (I think it was a mistake);
- I was trying to append second 4Tb drive to pool as second drive of the mirror, but apparently it wasn't already a mirror. I didn't understand how to do it.
- I found the button to expand pool to full size of 4Tb. I thought it would solve the problem with adding second drive to mirror, since pool was 2Tb, and I was trying to add 4Tb drive. But there was an error, and instead of expand to 4Tb, sole drive was disconnected from pool.
- I had the pool without any drive, and there was a suggestion to append disconnected 4Tb drive to pool as a new drive. But I didn't want to lose data. So I decided to import pool from 4Tb drive, since resilvering was successful.
- Button import showed no pools to import. So I decided to export the empty pool.
- Having no pools, import, the same way, showed no pool to import.

Now I have two 2Tb detached drives with the data, and one 4Tb drive, which wasn't detach, with the data, I hope.

Trying to use zpool import from any of three drives failed with an error: no pools available to import

Code:

bomber@truenas:~$ sudo /usr/sbin/zpool import -D -d /dev/sdb
no pools available to import

bomber@truenas:~$ sudo /usr/sbin/zpool import -D -d /dev/sdc
no pools available to import

bomber@truenas:~$ sudo /usr/sbin/zpool import -D -d /dev/sdd
no pools available to import

bomber@truenas:~$ sudo /usr/sbin/zpool import -D -d /dev/sdd1
no pools available to import

zdb also wasn't able to help:

Code:

bomber@truenas:~$ sudo /usr/sbin/zdb -l /dev/sdb
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

bomber@truenas:~$ sudo /usr/sbin/zdb -l /dev/sdc
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

bomber@truenas:~$ sudo /usr/sbin/zdb -l /dev/sdd
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

bomber@truenas:~$ sudo /usr/sbin/zdb -l /dev/sdd1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

sgdisk showed something interesting only for 4Tb drive:

Code:

bomber@truenas:~$ sudo /usr/sbin/sgdisk -p /dev/sdb
Creating new GPT entries in memory.
Disk /dev/sdb: 3907029168 sectors, 1.8 TiB
Model: WDC WD20EARX-00A
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): CE4B77D0-93F4-4143-A0A9-0FFFD905BBCD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3907029101 sectors (1.8 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name


bomber@truenas:~$ sudo /usr/sbin/sgdisk -p /dev/sdc
Creating new GPT entries in memory.
Disk /dev/sdc: 3907029168 sectors, 1.8 TiB
Model: WDC WD20EARX-00P
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): B2157331-9163-47FA-BC09-A4897FC80E9A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3907029101 sectors (1.8 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name


bomber@truenas:~$ sudo /usr/sbin/sgdisk -p /dev/sdd
Disk /dev/sdd: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 7BC873C3-B3BA-4776-BB3F-ECB3521CD9E7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      7814037134   3.6 TiB     BF01 


bomber@truenas:~$ sudo /usr/sbin/sgdisk -p /dev/sdd1
Creating new GPT entries in memory.
Disk /dev/sdd1: 7814035087 sectors, 3.6 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): A62B361C-3B24-4B41-BC3D-2A0E70621E67
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814035053
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814035020 sectors (3.6 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name

zpool list sees only boot drive:

Code:

bomber@truenas:~$ sudo /usr/sbin/zpool list -vL
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool    67G  22.8G  44.2G        -         -    10%    34%  1.00x    ONLINE  -
  sda3     67.3G  22.8G  44.2G        -         -    10%  34.0%      -    ONLINE

The magic with dd didn't show anything for both 2Tb detached drive. They apparently don't contain any lables anymore...

Code:

bomber@truenas:~$ sudo dd if=/dev/sdb bs=1M | strings | grep metaslab_shift
^C859+0 records in
858+0 records out
899678208 bytes (900 MB, 858 MiB) copied, 16.1243 s, 55.8 MB/s

bomber@truenas:~$ sudo dd if=/dev/sdc bs=1M | strings | grep metaslab_shift
^C810+0 records in
809+0 records out
848297984 bytes (848 MB, 809 MiB) copied, 14.7885 s, 57.4 MB/s

But there is something in 4Tb drive I found!

Code:

bomber@truenas:~$ sudo dd if=/dev/sdd bs=512 iseek=4096 count=2048 of=/tmp/img.sdd
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157901 s, 66.4 MB/s

bomber@truenas:~$ sudo /usr/sbin/zdb -l /tmp/img.sdd
------------------------------------
LABEL 0
------------------------------------
    version: 28
    name: 'Data'
    state: 0
    txg: 65964494
    pool_guid: 839021082166356816
    errata: 0
    hostid: 901624821
    hostname: 'truenas'
    top_guid: 6527413099654331064
    guid: 6527413099654331064
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6527413099654331064
        path: '/dev/disk/by-partuuid/3ac1a818-76a6-420f-be89-69fcc6a3d7c4'
        whole_disk: 0
        metaslab_array: 30
        metaslab_shift: 34
        ashift: 12
        asize: 2000394125312
        is_log: 0
        DTL: 4231
        create_txg: 4
    features_for_read:
    labels = 0 1
failed to unpack label 2
failed to unpack label 3

It's remnants of my pool! But I couldn't find offsets to find label 2 and 3.
Please help! How can I restore my data?
I use TrueNAS Scale 23.10.1

 

[RockBomber]

I still try to restore my disks and now I've found another two labels at the end of the disk:

Code:

$ sudo dd if=/dev/sda bs=512 iseek=7814034432 count=2048 of=/tmp/img.sda2

sudo /usr/sbin/zdb -l /tmp/img.sda2
------------------------------------
LABEL 0 (Bad label cksum)
------------------------------------
    version: 28
    name: 'Data'
    state: 0
    txg: 65964494
    pool_guid: 839021082166356816
    errata: 0
    hostid: 901624821
    hostname: 'truenas'
    top_guid: 6527413099654331064
    guid: 6527413099654331064
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6527413099654331064
        path: '/dev/disk/by-partuuid/3ac1a818-76a6-420f-be89-69fcc6a3d7c4'
        whole_disk: 0
        metaslab_array: 30
        metaslab_shift: 34
        ashift: 12
        asize: 2000394125312
        is_log: 0
        DTL: 4231
        create_txg: 4
    features_for_read:
    labels = 0 1
failed to unpack label 2
failed to unpack label 3

I think something went wrong when I pressed button to resize pool to maximum size of the disk.
Because the "asize" shows capacity about 2Tb, but another two labels locate at the end of 4Tb disk. And it's strange that they have named label 0 and label 1 instead of label 2 and label 3.

But I still cannot figure out what I can do to restore the pool.
Since I have two 4Tb disk I cloned this one to another in order to trying things.

Also I tried to apply command zhack laber repair -u to restore the label on a detached 2Tb, but it doesn't work

Code:

$ sudo /usr/sbin/zhack label repair /dev/sdb -u

Calculated filesize to be 2000398934016
error: label 0: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 1: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 2: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 3: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
label 0: uberblock: skipped checksum: skipped
label 1: uberblock: skipped checksum: skipped
label 2: uberblock: skipped checksum: skipped
label 3: uberblock: skipped checksum: skipped

 

[HoneyBadger]

Since I have two 4Tb disk I cloned this one to another in order to trying things.

@RockBomber I was about to ask you to do this as a preventative measure; happy to see that you have already done it.

Reviewing your post above it looks like a partition exists on the 4TB drive (and the clone as well?) - do you recall what version of TrueNAS you were on when you built your initial 2TB pool?

 

[RockBomber]

Thank you for the answer!
I initialed a 2Tb pool a long time ago on Nas4Free (XigmaNAS presently). Then successfully imported poll to TrueNAS Scale without upgrading the pool in order to have an opportunity to revert to XigmaNAS.
I don't precisely remember version of Nas4Free, it was presumably about 12 years ago :)

I've just carefully read the topic https://forums.freebsd.org/threads/recover-lost-mirror-in-a-zfs-pool.64462/
And I moved Label to the right place

Code:

sudo dd if=/dev/sda bs=512 iseek=4096 count=8192 of=/tmp/sda_header

sudo dd if=/tmp/sda_header bs=512 of=/dev/sda

Now zdb -l /dev/sda shows me label:

Code:

$ sudo /sbin/zdb -l /dev/sdb

------------------------------------
LABEL 0
------------------------------------
    version: 28
    name: 'Data'
    state: 0
    txg: 65964494
    pool_guid: 839021082166356816
    errata: 0
    hostid: 901624821
    hostname: 'truenas'
    top_guid: 6527413099654331064
    guid: 6527413099654331064
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6527413099654331064
        path: '/dev/disk/by-partuuid/3ac1a818-76a6-420f-be89-69fcc6a3d7c4'
        whole_disk: 0
        metaslab_array: 30
        metaslab_shift: 34
        ashift: 12
        asize: 2000394125312
        is_log: 0
        DTL: 4231
        create_txg: 4
    features_for_read:
    labels = 0 1
failed to unpack label 2
failed to unpack label 3

But importing failed yet:

Code:

$ sudo /sbin/zpool import -d /dev/disk/by-id/ata-WDC_WD40EFPX-68C6CN0_WD-WX12D83L4ULA

   pool: Data
     id: 839021082166356816
  state: FAULTED
status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
 config:

        Data                                        FAULTED  corrupted data
          ata-WDC_WD40EFPX-68C6CN0_WD-WX12D83L4ULA  ONLINE

I keep sifting though the internet to find solutions)

 

[HoneyBadger]

I've just carefully read the topic https://forums.freebsd.org/threads/recover-lost-mirror-in-a-zfs-pool.64462/
And I moved Label to the right place

I hope that was done on your clone 4TB?

You'll need to restore the original data again - a full clone shouldn't be necessary, just repeat the same commands but pull the section of data from your "good 4TB" and write it to your "clone 4TB" - for safety's sake, I'd suggest clearly labeling them as "Original" and "Clone" and only having one drive connected to the system at a time. Reading your comments in more detail it looks like you've also shifted the sectors a little (source was 8192 sectors at 4096 offset, but you wrote without an offset) so let's just copy more of the original:

Code:

# disconnect clone drive, connect original

sudo dd if=/dev/sda bs=512 count=12288 of=~/original.header

# disconnect original drive, connect clone

sudo dd if=~/original.header bs=512 of=/dev/sda

Once you've restored the original drive sectors please provide me with sfdisk -d /dev/sda where sda is the "clone disk"

 

[RockBomber]

I hope that was done on your clone 4TB?

Yeah, it was done on the clone of course)

I'd suggest clearly labeling them as "Original" and "Clone" and only having one drive connected to the system at a time.

Thank you! I always do exactly that way)

Code:

# disconnect clone drive, connect original
sudo dd if=/dev/sda bs=512 count=12288 of=~/original.header

# disconnect original drive, connect clone
sudo dd if=~/original.header bs=512 of=/dev/sda

Once you've restored the original drive sectors please provide me with sfdisk -d /dev/sda where sda is the "clone disk"

I restored original sectors, here the output:

Code:

$ sudo /sbin/sfdisk -d /dev/sdb

label: gpt
label-id: 7BC873C3-B3BA-4776-BB3F-ECB3521CD9E7
device: /dev/sdb
unit: sectors
first-lba: 34
last-lba: 7814037134
sector-size: 512

/dev/sdb1 : start=        2048, size=  7814035087, type=6A898CC3-1DD2-11B2-99A6-080020736631, uuid=3AC1A818-76A6-420F-BE89-69FCC6A3D7C4

 

[HoneyBadger]

Okay, let's try some magic.

sudo /sbin/sfdisk -d /dev/sdb > /tmp/magic.disk

This will dump that partition table to a file. Now you need to edit this file with nano or vi or your preferred text editor of choice, and change the start and size values in the last line (only change the last line) to read:

/dev/sdb1 : start= 4096, size= 7814031361, type=6A898CC3-1DD2-11B2-99A6-080020736631, uuid=3AC1A818-76A6-420F-BE89-69FCC6A3D7C4

Save the file, and then commit it to your cloned disk with

sudo /sbin/sfdisk /dev/sdb < /tmp/magic.disk

And then let's see what zpool import says.

 

[RockBomber]

Wow, it works! Pure magic!

Code:

$ sudo /sbin/zpool status

  pool: Data
 state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on software that does not support
        feature flags.
  scan: resilvered 372G in 01:16:32 with 0 errors on Sat Jan 20 17:28:47 2024
config:

        NAME                                    STATE     READ WRITE CKSUM
        Data                                    ONLINE       0     0     0
          3ac1a818-76a6-420f-be89-69fcc6a3d7c4  ONLINE       0     0     0

errors: No known data errors

Thank You very much!

I'll try to get back the pool to TrueNAS Scale and to add another disk to vdev as a mirror

 

[HoneyBadger]

I'll try to get back the pool to TrueNAS Scale and to add another disk to vdev as a mirror

Find your pool under the Storage and click the Manage Devices button. Then select your single 4TB disk and select Extend on the right-hand-side:

 

[RockBomber]

Thank You again!
That went smoothly too.
I'll wait for resilvering, and it seems like the last thing left to do is to say the pool use full capacity of 4Tb disk, since the size of the pool is only 1.7Tb

 

[HoneyBadger]

Thank You again!
That went smoothly too.
I'll wait for resilvering, and it seems like the last thing left to do is to say the pool use full capacity of 4Tb disk, since the size of the pool is only 1.7Tb

Once the resilver is done, I'd suggest doing a system reboot just to ensure that there aren't any weird artifacts left from the manual partition editing we had to do from the CLI. If it doesn't show as the correct 4T size after reboot then come back here and we'll check in on it.

 

[RockBomber]

Everything is fine after reslivering and rebooting. The pool's size 1.81T after rebooting.

Code:

$ sudo /sbin/zpool list

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Data       1.81T   371G  1.45T        -     1.81T      -    19%  1.00x    ONLINE  /mnt
boot-pool    67G  22.9G  44.1G        -         -    10%    34%  1.00x    ONLINE  -

The EXPANDSZ field suggests that the pool can be expanded.
There is the button "Expand" in the web UI, but last time I used it ends up with the very error you've just helped me to solve)

 

[HoneyBadger]

To circle back, for safety's sake (and avoiding the chance of something going amiss again) we did this one from the command line with zpool online -e Poolname GUID-OF-DEVICE against both of the disks.

Happy to have helped out!

 

[FrankWard]

Yeah it is, so, I disconnected one of the drives, that I was supposed to move to another computer and setup a backup on that machine.
Hovewer I took the wrong drive and reformatted and also have put some files on it, so that drive I really cant do anything with.

I must have done something I cant remember, I was in a process swapping drives etc to organize..
Yeah I know, I do have that.. And I checked the serial before removing, but I was a little distracted while doing so.. and somehow mixed the seirals.

It's a good idea to figure out serials and other identifying data for the drive and LABEL THE DRIVES before you put them into the machine. This way you be sure you're pulling the correct disk when you open the box.