pool locked, no unlock

[gigogig]

We have an encrypted pool that we cant unlock, and to the best of my knowledge its not from any of the usual ways.
In the ui (Version: FreeNAS-11.3-U2.1 ) the 110 disks are listed (although missing pool name and disk size) in disks.
in the pool section the pool is listed as locked.
the options that are available to the pool:
export/disconnect - [ENOENT] None: PoolDataset poolname does not exist
extend - did not try
scrub pool - did not try as its locked
status - brings us to the Pool Status page with nothing listed

recovery key - brings us to a "Recovery Key for Pool poolname" page with two options add recovery key, invalidate existing key, nether seemed like a good idea so we went no further.
reset keys - also did not go into this as its locked and this seems like a bad idea.


but we don't have any options to use a recovery key or to be able to export / import the pool. any ideas?

 

[PhiloEpisteme]

So, a few things come to mind.

First, did you ever set a passphrase for your pool? What is the output of sqlite3 /data/freenas-v1.db 'select * from storage_volume;' and sqlite3 /data/freenas-v1.db 'select * from storage_encrypteddisk;'.

Second, how long have you been on 11.3-U2.1? And before that; were you on 11.3-Release, 11.3-U1, or 11.3-U2?

Also, did you do anything with your system that caused it to lock before you were unable to unlock it?

As a quick sanity check, can you try the following commands and give me the output? gpart show, ls -1 /dev/gptid, zpool status? If the output is extremely long feel free to attach it as a text file. If you paste it directly into the chat please include code tags like this

[code]
paste here
[/code]

 

[gigogig]

First, did you ever set a passphrase for your pool? no
Second, how long have you been on 11.3-U2.1? And before that; were you on 11.3-Release, 11.3-U1, or 11.3-U2?
started fresh (i think) FreeNAS-11.3-U1 on 2-25 (brand new pool)
~4-29 had to shut down the pool as power was being played with and we dident want a hard shutdown.
when we turned the system back on, but we forgot one of the trays the first time
so it ran for about a day or two and obviously could not be mounted
we turned the freenas box back off and powered on the last tray and powered back on the freenas box
but at that point we had the options for the pool i listed above.
so thinking this may have been just a bug with the version of freenas that we were on, we decided to upgrade to FreeNAS-11.3-U2.1
but after the upgrade the system could not find its boot volume.
so we did a fresh install (wiping the boot volume when prompted) and keeping the config
but we are still in the same place we started from now.
we do have the config backups with "Export Pool Encryption Keys " as well as the recovery.key and the encryption.key

 

[PhiloEpisteme]

Interesting, thanks for all that info. The reason I asked about what version you were on is because 11.3-U2.1 fixes a bug which can result in completely wiping a pool via destroying the partitions and the geli metadata. Looking at your data it looks like you were not hit by this bug.

I parsed your data and it looks like all of the drives that your system expects to be in the encrypted pool are already unlocked. I see this by comparing the UUID from those disks identified in the storage_encrypteddisk vs those providers in /dev/geli and found that every UUID in the database has a provider appended with .eli. You should double-check my work though.

If they are actually unlocked I would expect the pool to be listed in the output of zpool import. What does that show you? If that shows your pool there and in good health I would expect a restart to solve your problem; if it doesn't you'll want to search logs in /var/log for geli or zpool import commands failing.

You can also double-check your system's encryption key. Given that it seems that your drives are unlocked I would expect this not to solve your problem, but is worth investigating anyway. From the output you gave from storage_volume you should expect your pool's key to be found in /data/geli/813df0ec-5522-47cb-9ec6-83727677add8.key. If there is a file there you should diff it vs your encryption.key. If they are the same; this likely isn't your issue. If they are different, keep a copy of the file already in /data/geli and replace it with encryption.key.

Because you don't have a passphrase set you may need to reboot your machine. For encrypted pools without a passphrase FreeNAS goes through the unlock/import routine at startup.

Another thing worth checking is that the key your system is using, the one in /data/geli actually works for your drives. You can test this by either locking and then trying to unlock your drives geli detach /dev/gptid/<uuid>; geli attach -p -k /data/geli/<key>.key /dev/gptid/<UUID>. There is also a trick which you can use without locking the drive; try unlocking it while it is already unlocked. If you have the correct key you will get an error like geli: cannot access gptid/<UUID> (error=1). Also note that the UUID should NOT include the .eli suffix. You can get clever here and modify the sqlite3 query to return only the UUIDs, pipe that to xargs, and automatically execute the attach command for each to check every disk.

I wrote a resource about how to debug encryption issues; you can find it here.

 

[gigogig]

if you dont mind what vern / versions had this "wiping a pool via destroying the partitions and the geli metadata "? (have a system that's FreeNAS-11.3-U1 but due to my last update failing i'm nervous about updating it)

your zpool import idea seems to be on the mark (must be why i cant unlock the pool from the ui)

humm looking into "FAULTED corrupted data" (new to me) before trying a force mount

Code:

root@v-backup[~]# zpool import
   pool: BackupV
     id: 10103775338767589468
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-72
 config:

        BackupV                                             FAULTED  corrupted data
          raidz3-0                                          FAULTED  corrupted data
            gptid/44621786-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45bb039e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/454b98b3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45dd1a4b-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45ed50ac-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45e912ca-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45a1de1c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/45cd0b2d-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-1                                          ONLINE
            gptid/74a21caf-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/749ede47-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/74e54f9c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/755b94c7-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/7545c9c3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/756f318e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/757260e4-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/757de231-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-2                                          ONLINE
            gptid/b2d173a4-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b2e053fc-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b2f329f5-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b37da70d-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b37fb0d3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b3859ce1-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b3b06a5e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/b3a4c98c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-3                                          ONLINE
            gptid/e8c20ff8-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e9c8ea19-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea3fa5ab-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea6f4ed0-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea2aed56-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea5635c2-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea5bd0a7-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/ea6cf90c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-4                                          ONLINE
            gptid/16a3a118-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/16bd2645-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/15b06956-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/16a5ac92-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/162bab5c-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/164fde16-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/168cef09-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/16c0c253-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-5                                          ONLINE
            gptid/58da15b3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/58b7f49d-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/58ce4cf6-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/58cdba5d-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/58eed2df-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/5926779a-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/5921fad3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/58f0dad5-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-6                                          ONLINE
            gptid/e168a29f-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1c73212-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1c4bf5a-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1b1a477-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1d68ae3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1d98e9f-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e20bf114-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/e1fc1acc-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-7                                          ONLINE
            gptid/095836d9-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/093b19bd-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/098b2167-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/098b4a3c-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/0951bafa-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/097b7ec8-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/0951e3cb-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/09644950-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-8                                          ONLINE
            gptid/3a407cde-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3a7d0b0f-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3ab08a00-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3ab73b03-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3aa1ee14-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3a79607e-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3ab0b483-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/3ae50d66-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-9                                          ONLINE
            gptid/68db5743-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/68f21872-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/696d7111-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/69201aeb-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/69560f51-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/697b2695-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/698777b6-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/697740aa-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-10                                         ONLINE
            gptid/9bbe6034-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c0ace6f-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c7e5480-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c1b9b09-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c14f7bc-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c26427e-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c2618f3-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/9c677c40-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
          raidz3-11                                         ONLINE
            gptid/c77bf53d-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c757b704-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c770eb16-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c7be46cd-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c7a97d9a-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c75ee2b4-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c7879e83-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
            gptid/c770c1ca-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE
        cache
          gptid/b0b18b80-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0973d3a-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0a0dcd3-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b09e610f-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0c266b8-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0c0dd7f-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0ae94f7-5e73-11ea-a17f-b8ca3af50009.eli
          gptid/b0c04720-5e73-11ea-a17f-b8ca3af50009.eli
        spares
          gptid/5e6fedd0-89b1-11ea-8e5a-b8ca3af50009.eli
          gptid/5eae4b45-89b1-11ea-8e5a-b8ca3af50009.eli
          gptid/5ea4d21b-89b1-11ea-8e5a-b8ca3af50009.eli
          gptid/5ed57c10-89b1-11ea-8e5a-b8ca3af50009.eli
          gptid/5ed76156-89b1-11ea-8e5a-b8ca3af50009.eli

 

[PhiloEpisteme]

if you don't mind what vern / versions had this "wiping a pool via destroying the partitions and the geli metadata "? (have a system that's FreeNAS-11.3-U1 but due to my last update failing i'm nervous about updating it)

All versions of 11.3 prior to U2.1.

Yeah, I cannot offer more than the suggested force mount here. I believe there is a no-op flag as well which may tell you if it would be mountable with the -F without actually trying to do it.

Also, if you get it imported via CLI you should then export it via CLI and restart the machine so that the system will import it correctly for the GUI to work properly.

Also, what kind of drives are all of those cache drives? :)

 

[gigogig]

Thank you for your help PhiloEpisteme, anything i should do to prepare a FreeNAS-11.3-U1 install for an update (never had one fail before so still spooked)?

Yeah, I cannot offer more than the suggested force mount here. I believe there is a no-op flag as well which may tell you if it would be mountable with the -F without actually trying to do it.
ill try a -f shortly hopefully it can fix itself

Also, if you get it imported via CLI you should then export it via CLI and restart the machine so that the system will import it correctly for the GUI to work properly.
will do, but out of curiosity why the cli and not the ui?

Also, what kind of drives are all of those cache drives?
old 200gb hgst sas drive HUSMM8020ASS200 i think or an older sas 6 equivalent

 

[Yorick]

ill try a -f shortly

-F is the one that attempts to repair your pool.

anything i should do to prepare a FreeNAS-11.3-U1 install for an update

Take a backup of your config and all encryption keys, keep it outside the FreeNAS unit, make sure the backup is readable and secure.
Delete old boot environments if you think you might be running out of space.
Don't boot from a USB3 key; hopefully your boot media is SSD. USB2 may well survive as well, that's probably going to be okay. But, if you are currently booting from USB, have replacement boot media ready. SSD preferred. Only because USB sticks tend to die, faster if they are USB3.

 

[gigogig]

Sorry yes -F not -f
imported, online and No known data errors ?
going to do a scrub before i export confused what the problem was / is?

Take a backup of your config and all encryption keys, keep it outside the FreeNAS unit, make sure the backup is readable and secure.
always good advice
Delete old boot environments if you think you might be running out of space.
not a problem for us
Don't boot from a USB3 key; hopefully your boot media is SSD. USB2 may well survive as well, that's probably going to be okay. But, if you are currently booting from USB, have replacement boot media ready. SSD preferred. Only because USB sticks tend to die, faster if they are USB3.
unfortunately we are using some sandisk usb3 drives in usb2 ports (non have failed yet) and they are larger 32gb or more each and they are mirrored so with luck we will be ok

Code:

root@v-backup[~]# zpool import  BackupV -F
root@v-backup[~]# zpool status
  pool: BackupV
 state: ONLINE
  scan: scrub repaired 0 in 0 days 03:02:39 with 0 errors on Sun Apr 12 03:02:54 2020
config:

        NAME                                                STATE     READ WRITE CKSUM
        BackupV                                             ONLINE       0     0     0
          raidz3-0                                          ONLINE       0     0     0
            gptid/44621786-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45bb039e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/454b98b3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45dd1a4b-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45ed50ac-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45e912ca-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45a1de1c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/45cd0b2d-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-1                                          ONLINE       0     0     0
            gptid/74a21caf-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/749ede47-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/74e54f9c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/755b94c7-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/7545c9c3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/756f318e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/757260e4-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/757de231-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-2                                          ONLINE       0     0     0
            gptid/b2d173a4-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b2e053fc-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b2f329f5-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b37da70d-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b37fb0d3-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b3859ce1-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b3b06a5e-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/b3a4c98c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-3                                          ONLINE       0     0     0
            gptid/e8c20ff8-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e9c8ea19-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea3fa5ab-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea6f4ed0-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea2aed56-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea5635c2-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea5bd0a7-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/ea6cf90c-5e72-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-4                                          ONLINE       0     0     0
            gptid/16a3a118-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/16bd2645-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/15b06956-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/16a5ac92-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/162bab5c-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/164fde16-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/168cef09-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/16c0c253-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-5                                          ONLINE       0     0     0
            gptid/58da15b3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/58b7f49d-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/58ce4cf6-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/58cdba5d-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/58eed2df-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/5926779a-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/5921fad3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/58f0dad5-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-6                                          ONLINE       0     0     0
            gptid/e168a29f-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1c73212-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1c4bf5a-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1b1a477-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1d68ae3-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1d98e9f-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e20bf114-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/e1fc1acc-5e73-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-7                                          ONLINE       0     0     0
            gptid/095836d9-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/093b19bd-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/098b2167-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/098b4a3c-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/0951bafa-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/097b7ec8-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/0951e3cb-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/09644950-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-8                                          ONLINE       0     0     0
            gptid/3a407cde-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3a7d0b0f-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3ab08a00-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3ab73b03-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3aa1ee14-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3a79607e-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3ab0b483-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/3ae50d66-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-9                                          ONLINE       0     0     0
            gptid/68db5743-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/68f21872-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/696d7111-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/69201aeb-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/69560f51-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/697b2695-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/698777b6-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/697740aa-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-10                                         ONLINE       0     0     0
            gptid/9bbe6034-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c0ace6f-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c7e5480-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c1b9b09-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c14f7bc-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c26427e-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c2618f3-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/9c677c40-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
          raidz3-11                                         ONLINE       0     0     0
            gptid/c77bf53d-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c757b704-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c770eb16-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c7be46cd-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c7a97d9a-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c75ee2b4-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c7879e83-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
            gptid/c770c1ca-5e74-11ea-a17f-b8ca3af50009.eli  ONLINE       0     0     0
        cache
          gptid/b0b18b80-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0973d3a-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0a0dcd3-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b09e610f-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0c266b8-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0c0dd7f-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0ae94f7-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
          gptid/b0c04720-5e73-11ea-a17f-b8ca3af50009.eli    ONLINE       0     0     0
        spares
          gptid/5e6fedd0-89b1-11ea-8e5a-b8ca3af50009.eli    AVAIL
          gptid/5eae4b45-89b1-11ea-8e5a-b8ca3af50009.eli    AVAIL
          gptid/5ea4d21b-89b1-11ea-8e5a-b8ca3af50009.eli    AVAIL
          gptid/5ed57c10-89b1-11ea-8e5a-b8ca3af50009.eli    AVAIL
          gptid/5ed76156-89b1-11ea-8e5a-b8ca3af50009.eli    AVAIL

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            da150p2  ONLINE       0     0     0
            da152p2  ONLINE       0     0     0

errors: No known data errors

 

[Yorick]

we are using some sandisk usb3 drives in usb2 ports

I am told that'll work just as well as USB2 sticks. Does your board support SATADOM? That may be another option. Not right this second, obviously, but something to look at to improve the resilience of the boot media.

 

[PhiloEpisteme]

Sorry yes -F not -f
imported, online and No known data errors ?
going to do a scrub before i export confused what the problem was / is?

I'm a bit perplexed as well. Perhaps something went wrong with one of the last transaction groups? Either way; the fact that it imported is a good sign! Smart idea to run the scrub.

 

[Apollo]

TLTR: Makes me think that maybe the Backup pool was created through replication, if so, if the pool has never been mounted because the pool would be set as Read only, then the content of the pool will not be made visible by the system.
Listing the snapshot will indicate whether the pool contains data.

If the above is the issue, seting the readonly flag to off and reboot the system should solve the issue.