HDD not be remembered by pool? Unable to access data (Half a TB's)

[Aden MGB]

Hello people who I hope can help

Okay so a bit of context I am running TrueNAS core on an upcycled all in one (not ideal I know but better then E-Waste). I have one 1TB WD blue hard which is striped (once again I know its not ideal but it's better then nothing), also I had attempted and update to a later version of TrueNAS core but it failed for some reason (?) anyways so I did have a configuration backup.

Okay so in this upcycled machine there is only one SATA port so I tested adding a USB to the pool (really should have used a separate pool now that I think about it) because I was planning to get an external USB HDD for it, it works I remove the drive and suddenly the CPU hits 75% usages (A 4 core 4 thread AMD chip) so I let it be for a while I thought maybe it's trying to reconstruct the data (I didn't put any on the USB) anyways sometime later its still not calming down so I restart the machine, there was no warning messages of any kind so I thought it would be okay, sometime later I was about to watch Plex but it said that the server was inaccessible which I thought was weird considering the server is on, I log onto the web interface and low and behold it says the the pool "Main" is completely offline.


Instinctively I check the disk that the system sees and the HDD is still there (?) same serial everything, I even ran tests on it and it reported fine. So with the knolodge that I had a config backup saved locally I clicked the only option for the pool which was "Export/Disconnect" and I did that making sure not to erase the drive. I tried re-importing it, and it didn't recognize anything (?) I tried decrypting it but once again it didn't show up so now I have taken out the HDD and the boot drive, may I add that the boot pool is still fine.

So yeah if you guys know anything that would be great

 

[jgreco]

The middleware can't cope with duplicated serial numbers. This is part of how it identifies multipath disks in a SAS topology. TrueNAS does not support using USB disks in your data pool. See the excellent resource by @Arwen

Why you should avoid USB attached drives for data pool disks

This subject has been coming up quite often in the last year or 2. Perhaps because of TrueNAS SCALE has brought forth some attention to TrueNAS in general and ZFS in specific. Please note that this is about USB attached storage for ZFS data...

www.truenas.com

 

[Aden MGB]

The middleware can't cope with duplicated serial numbers. This is part of how it identifies multipath disks in a SAS topology. TrueNAS does not support using USB disks in your data pool. See the excellent resource by @Arwen

Why you should avoid USB attached drives for data pool disks

This subject has been coming up quite often in the last year or 2. Perhaps because of TrueNAS SCALE has brought forth some attention to TrueNAS in general and ZFS in specific. Please note that this is about USB attached storage for ZFS data...

www.truenas.com

Hmmm that's unfortunate, is there a way I could still get the data off the HDD and put it onto another machine? Or is it a complete lost situation, because from my knowledge I don't believe the HDD was encrypted?

 

[jgreco]

You should be able to get the pool imported but you'll probably have to do it via the command line. I don't do this often enough that I have specific guidance off the top of my head.

 

[Aden MGB]

Would you know the command? because the 'zpool import' command isn't working even though it should recognize the pool from the disk?

The system clearly recognizes the disk but is there any way that I could import it into another dataset without the data being erased as it already in the ZFS filesystem?

 

[HoneyBadger]

Hello @Aden MGB

When you added the test USB device to your system, it's likely that using the regular expansion method caused your data to stripe across the HDD and your USB device. If you removed the USB device physically without using the "Remove vdev" functionality within ZFS, you're likely looking at a stripe pool with a missing device, which won't cleanly import and may have missing data.

Can you enable SSH access (including "root access with password") from the Services page, connect using a client such as PuTTY, and post the output of the zpool import inside of [CODE][/CODE] tags?

 

[Aden MGB]

Code:

FreeBSD 13.1-RELEASE-p7 n245428-4dfb91682c1 TRUENAS

        TrueNAS (c) 2009-2023, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# zpool import
no pools available to import
root@truenas[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool    29G  1.30G  27.7G        -         -     0%     4%  1.00x    ONLINE  -
root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# glabel status
                                      Name  Status  Components
gptid/ede97451-0a94-11ee-bff7-dcfe07d40a55     N/A  da0p1
root@truenas[~]# 

Hopefully the code thing worked properly but as you can see it doesn't even recognize it, although I don't fully remember how but when I used to use the

Code:

zpool list

command it would list the pool "Main" but it was say failed to import or something along the lines of that because it stated "ONE OR MORE VDEVS UNVAIL" which is to be expected as I did remove a drive but the pool wouldn't let me access the data at all and I read online that it should be still accessible so I don't really know what to think of that. Anyways I hope this code helps @HoneyBadger

 

[Aden MGB]

Oh wait I have noticed my notifications on the web interface have told me something,

maybe this could tell us something

 

[winnielinnie]

You might have inadvertently destroyed your Main pool (as explained by the others above.)

If you really need this data, and you don't have any other copies or backups, then your last ditch option is to use a data recovery software or service.

Klennet ZFS Recovery is one such option.

There's also (the not user-friendly) Photorec, which is part of the Testdisk suite. It's a terminal-based application that can recover sequentially stored files based on known "signatures"; even if the filesystem is corrupt or damaged. However, this means if it does recover files, you will lose their original filenames and timestamps. (There are exceptions, such as parsing "useable info" from certain filetypes, such as images and documents. If the file itself contains embedded metadata, it can try to use this as a makeshift filename or timestamp.)

The name "Photorec" is a misnomer now, since it can recover much more than just photos.

 

[HoneyBadger]

Hopefully the code thing worked properly but as you can see it doesn't even recognize it, although I don't fully remember how but when I used to use the

Code:

zpool list

command it would list the pool "Main" but it was say failed to import or something along the lines of that because it stated "ONE OR MORE VDEVS UNVAIL" which is to be expected as I did remove a drive but the pool wouldn't let me access the data at all and I read online that it should be still accessible so I don't really know what to think of that. Anyways I hope this code helps @HoneyBadger

I was expecting a readout similar to below, but with references to UNAVAIL and "missing" vs the example.

Code:

root@core01[~]# zpool import
   pool: stripetest
     id: 16583979732924573554
  state: UNAVAIL
status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

        stripetest                                    UNAVAIL  insufficient replicas
          gptid/58c05e23-d3c6-11ed-86d3-000c2994688a  ONLINE
          gptid/58c25058-d3c6-11ed-86d3-000c2994688a  FAULTED  corrupted data

If you attempt to import via zpool import Main and get the error related to missing devices, then you likely created a stripe as shown above. The ideal solution is "restore the data from backups" but if you have no other backups and are willing to risk data loss, you can attempt a recovery via the following steps.

Show : Here Be Dragons

From the SSH prompt, try zpool import -F Main -o readonly=on

If this fails, you can attempt to import with a missing vdev. As per some warnings from ZFS, this option can be extremely hazardous to the health of your pool and should only be used as a last resort.

Show : Seriously Large Dragons Now

The -X option, when used with the -F recovery option, determines whether "extreme measures" to find a valid txg should take place. This allows the pool to be rolled back to a txg which is no longer guaranteed to be consistent. Pools imported at an inconsistent txg may contain uncorrectable checksum errors.

From the SSH prompt, enter sysctl vfs.zfs.max_missing_tvds=1 and then enter zpool import -FX Main -o readonly=on

If at any point the pool imports (in read-only mode) then immediately begin copying critical data off to another location.[/code]

 

[Aden MGB]

I was expecting a readout similar to below, but with references to UNAVAIL and "missing" vs the example.

Code:

root@core01[~]# zpool import
   pool: stripetest
     id: 16583979732924573554
  state: UNAVAIL
status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

        stripetest                                    UNAVAIL  insufficient replicas
          gptid/58c05e23-d3c6-11ed-86d3-000c2994688a  ONLINE
          gptid/58c25058-d3c6-11ed-86d3-000c2994688a  FAULTED  corrupted data

If you attempt to import via zpool import Main and get the error related to missing devices, then you likely created a stripe as shown above. The ideal solution is "restore the data from backups" but if you have no other backups and are willing to risk data loss, you can attempt a recovery via the following steps.

Show : Here Be Dragons

From the SSH prompt, try zpool import -F Main -o readonly=on

If this fails, you can attempt to import with a missing vdev. As per some warnings from ZFS, this option can be extremely hazardous to the health of your pool and should only be used as a last resort.

Show : Seriously Large Dragons Now

The -X option, when used with the -F recovery option, determines whether "extreme measures" to find a valid txg should take place. This allows the pool to be rolled back to a txg which is no longer guaranteed to be consistent. Pools imported at an inconsistent txg may contain uncorrectable checksum errors.

From the SSH prompt, enter sysctl vfs.zfs.max_missing_tvds=1 and then enter zpool import -FX Main -o readonly=on

If at any point the pool imports (in read-only mode) then immediately begin copying critical data off to another location.[/code]

That is very helpful so I will try that later, although right now thanks to your advice when I have time tommorow on the weekend I will use PhotoTec recovery software to try and recover the files first before I try the risky command, but thank you so much,

Thanks, Aden

 

[winnielinnie]

If you have a large enough spare drive, you can make an "image" of your device to hold onto temporarily, just in case you want to try recovery options on the image itself.

 

[Aden MGB]

The -X option, when used with the -F recovery option, determines whether "extreme measures" to find a valid txg should take place. This allows the pool to be rolled back to a txg which is no longer guaranteed to be consistent. Pools imported at an inconsistent txg may contain uncorrectable checksum errors.

From the SSH prompt, enter sysctl vfs.zfs.max_missing_tvds=1 and then enter zpool import -FX Main -o readonly=on

I tried to use that but this is what it outputted:

Code:

root@truenas[~]# sysctl vfs.zfs.max_missing_tvds=1
vfs.zfs.max_missing_tvds: 1 -> 1
root@truenas[~]# zpool import -FX Main -o readonly=on
cannot import 'Main': no such pool available
root@truenas[~]# 

 

[HoneyBadger]

I tried to use that but this is what it outputted:

Code:

root@truenas[~]# sysctl vfs.zfs.max_missing_tvds=1
vfs.zfs.max_missing_tvds: 1 -> 1
root@truenas[~]# zpool import -FX Main -o readonly=on
cannot import 'Main': no such pool available
root@truenas[~]# 

Assuming your pool is named "Main" it should have attempted the import at least. The fact that it isn't recognizing your pool at all (edit) isn't promising.

You can attempt to import a pool by device by using the -d argument and pointing to a device such as zpool import -FX -d /dev/ada0

 

[Aden MGB]

Assuming your pool is named "Main" it should have attempted the import at least. The fact that it isn't recognizing your pool at all

You can attempt to import a pool by device by using the -d argument and pointing to a device such as zpool import -FX -d /dev/ada0

I tried it and it almost like the device has forgotten it's pool because this was the output that came out when I tried

Code:

root@truenas [~]# zpool import -FX -d /dev/ada0 no pools available to import root@truenas [~]#

 

[HoneyBadger]

I tried it and it almost like the device has forgotten it's pool because this was the output that came out when I tried

Code:

root@truenas [~]# zpool import -FX -d /dev/ada0 no pools available to import root@truenas [~]#

What do you get as output from zdb -l /dev/ada0p2 ?

 

[jgreco]

I tried it and it almost like the device has forgotten it's pool because this was the output that came out when I tried

ada0 is not a valid device name for ZFS on FreeBSD. Try ada0p2.

 

[Aden MGB]

What do you get as output from zdb -l /dev/ada0p2 ?

 

[HoneyBadger]

Assuming ada0 is still your WD 1TB drive, this isn't good.

What does gpart status show? It should look similar to:

Code:

  Name  Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
 da0p1      OK  da0
 da0p2      OK  da0

 

[Aden MGB]

Assuming ada0 is still your WD 1TB drive, this isn't good.

What does gpart status show? It should look similar to:

Code:

  Name  Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
 da0p1      OK  da0
 da0p2      OK  da0

this is defiantly interesting it appears gpart isn't recognizing the disk but TrueNAS is