Unable to unlock Pools after upgrade 11.2-u7 to 11.3-RELEASE

faust7th

Cadet
Joined
Jan 31, 2020
Messages
4
Hello.

I just now updated from 11.2 to 11.3, and now i can not unlock my encrypted raidz Pool anymore.
Passphrase and recovery key are not working anymore.

The message i receive is just an "Error Unlocking"

Is there anything i can do or try to access my data again?
Thanks for you help.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Your best option would be to boot from 11.2
You can do that from the "Boot" section under 11.3 and activate the previous boot environment or restart your system and select the boot then.
Then you would need to remove the passphrase from the pool and then return to 11.3.
 

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
654
@faust7th You can try to unlock them manually from console but be very careful doing so! (Stop any SMB/NFS services for safety reasons)

You will need the encryption hey and GPTIDs. The key i assume you have. For the GPTIDs check output of glabel status which will gives you mapping against (a)daXp2. You always need <whatever>p2 (unless you've ripped off the swap). If you still don't know which disks are the encrypted ones you can check your backup config (assuming you have one from before upgrade). Just sqlite3 /path/to/cfgbackup.db then call this SQL:
Code:
SELECT storage_volume.vol_name, storage_encrypteddisk.encrypted_provider from storage_volume JOIN storage_encrypteddisk ON storage_volume.id = storage_encrypteddisk.encrypted_volume_id;

It will gives you pool name and GPTID of the related devices.

Now that you have both pieces you can try following:
Code:
geli attach -k /path/to/encryptionkey.key /dev/gptid/0faaxxxxxx-xxxxx-xxx-xxxx-xxxxxxxxxxxxx
geli attach -k /path/to/encryptionkey.key /dev/gptid/0fbbxxxxxx-xxxxx-xxx-xxxx-xxxxxxxxxxxxx

repeat for each encrypted volume. It will ask for passphrase for each unlock unless you're using recovery key.

If you manage to unlock all of the volumes you can then try to import
Code:
zpool import
zpool import poolname


Since you're bypassing the GUI/middleware the mount-point would be most probably broken so you can fix it like this:
Code:
zfs set mountpoint=/mnt/poolname poolname
zfs mount -a


Please note that this is NOT permament solution of your problem. It is more like emergency way how to get to your data and copy them elsewhere from console. If you now start messing with the pool/volume from within GUI (which is not aware that the pool is unlocked/mounted) it can have unexpected or even destructible consequences. You've been warned.

Once you're done with data backup you can try to zpool export poolname, restart the NAS and try to import it from within the GUI. Or you can just destroy the whole pool, re-create it from scratch and copy the data back.

Side-Note: If the manual unlock failed then you will get appropriate error message. Either your key/passphrase is invalid or the data on the disks got corrupted/overwritten. In case the geli metadata got corrupted and you have backup of these you might be able to restore them. Yet this is something unusual so i will not post details there as one rushed mistake would lock you out of your data forever...
 
Last edited:

faust7th

Cadet
Joined
Jan 31, 2020
Messages
4
[SOLVED]
Thanks for the answers.

Unlocking the Pool from the command line did not work for me and due to my own stupidity the old boot environment was not saved.

What I did:
  • I had a FreeNAS config backup from before the upgrade
  • I Installed the old FreeNAS Version again (11.2-U17 ... i think)
  • Uploaded the old config.
  • Unlocked the pool and removed the Passphrase (as sugested)
  • Started the Upgrade to 11.3-STABLE again
  • Pool was accessible this time.
Conclusion .... DON'T upgrade with a passphrase protected pool.
Thanks a lot Apollo and HolyK
 

SPo

Cadet
Joined
Mar 1, 2020
Messages
4
Please tell me, how you removed the passphrase - do i need to rekey the pool with a new key and without a passphrase ? Or do i have to set e new empty passphrase - and do i have to make a new recovery key then?

I cannot find the option to delete the passphrase in the help.

Thank you, SPo
 

faust7th

Cadet
Joined
Jan 31, 2020
Messages
4
The answer to all of it is "yes".

You have to rekey it with an empty passphrase and (I think) you have to make a new recovery key. (at leased I did)
 

SPo

Cadet
Joined
Mar 1, 2020
Messages
4
hi faust7th,

thx for the reply. After digging in the forum i found the solution: ther remove passphrase is only available in the old webui from 11.2.x. Ok - now for others that need help:


At first disconnect all unused disks in pools, because it will confuse you in step 12. Then use the following procedure:

1. go in Freenas GUI to "SYSTEM\BOOT" and activate the last 11.x (mine was 11.2-U8) for next boot.

2. Reboot the system.

3. Log into freenas but use the OLD webinterface!

4. Go to the volume overview and select your with passphrase encrypted volume

5. press in the bottom toolbar a button called new passphrase

6. enter your admin password

7. and select the checkbox "remove passphrase"

8. DONT reboot
after its done like this for all encrypted pools i added new recovery keys via the new webgui for each
pool and stored them - just to be safe. But i didnt need them.
VERY IMPORTANT:I also download the encryption key for each pool.

Then switch back to the 11.3 System:
9. go in Freenas GUI to "SYSTEM\BOOT" and activate the 11.3

10. Reboot the system.

11. got to "STORAGE\POOLS\" and export ONE POOL at a time AND DONT CHECK "Delete configuration of shares that used this pool?" or "Destroy data on this pool? ", just check confirm and click the button export/disconnect.

12. now add the pool via the upper right ADD button and check "import an existing pool". Then select all disks currently not used in other pools and add the pool. Because you didnt delete the settings in step 11, everything should now be fine - no extra work needed.
13. upload the encryption key, which you stored in step 8.

14. repeat step 11 and 13 for each of your pools.

15. reboot - pray - and all pools come up correctly and should be decrypted, because you removed the passphrase. I dunno how to reactivate the passphrase, but i am currently happy that i am on 11.3 with all my pool.

Bye and nice sunday!

SPo
 

SPo

Cadet
Joined
Mar 1, 2020
Messages
4
here the correct namings in old web ui:

4. Go to "Storage\Volumes\View Volumes" and select your with passphrase encrypted volume
5. press in the bottom toolbar the button "Change Passphrase"
 

hungarianhc

Patron
Joined
Mar 11, 2014
Messages
234
Is there a bug filed for this? I just upgraded my system to 11.3, and I had the exact same issue. Rolled back to 11.2, but that 15 step process seems a little scary... I think I'll stay on 11.2 for a bit. The issue is that when you roll back to 11.2, your jails don't work anymore. UGH!
 

SPo

Cadet
Joined
Mar 1, 2020
Messages
4
hi hungarianhc,

don`t be overwhelmed by the amount of steps - it is really straight forward and very easy. I tend to be very precise with the description of the procedure, so that it`s easy to follow all steps. Nevertheless the downside are 15 steps.:p

If you make a backup before (which, of course, is already done by you and up to date;) ) all steps in sum are a riskless procedure and you are afterwards on the newest version.

I did not file a bug - sry, never thought about it after a workaround was found for this one-time-procedure.

The ZFS needs to be upgraded afterwards, too - some parameters where added in newest version. But it will be announced on the gui:

WARNING
New feature flags are available for volume ssdpool. Refer to the "Upgrading a ZFS Pool" subsection in the User Guide "Installing and Upgrading" chapter and "Upgrading" section for more instructions.

SPo
 

troun

Dabbler
Joined
Jul 13, 2013
Messages
33
hi hungarianhc,

don`t be overwhelmed by the amount of steps - it is really straight forward and very easy. I tend to be very precise with the description of the procedure, so that it`s easy to follow all steps. Nevertheless the downside are 15 steps.:p

If you make a backup before (which, of course, is already done by you and up to date;) ) all steps in sum are a riskless procedure and you are afterwards on the newest version.

I did not file a bug - sry, never thought about it after a workaround was found for this one-time-procedure.

The ZFS needs to be upgraded afterwards, too - some parameters where added in newest version. But it will be announced on the gui:



SPo

Hey SPo, would you mind to fill a bug report please.
Short story long story, I had problem both on main server and backup, I was able to solve it for the backup following this post but after 3 days trying to get my main server up, still no success. So my only hope so far is to have an update solving it. (or resynchronise several TB over internet...)
 

rio

Cadet
Joined
May 4, 2020
Messages
3
Debug attached. The situation now is that I removed the passphrase while running 11.2 per SPo's suggestion, but I have to use the saved geli.key to unlock under 11.3.
I'm reluctant to export or detach and reimport since the documentation says, in bold in a warning block, "An encrypted pool cannot be reimported without a passphrase!".
Before removing the passphrase, there wasn't even a way to try to unlock it under 11.3: the web interface seemed broken, couldn't expand "Storage" or some of the other left menu sections and nothing happened when clicking anywhere in the "Pool" box in the dashboard view.
 

Attachments

  • debug-freenas-20200506085240.tgz
    437.9 KB · Views: 292

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Debug attached. The situation now is that I removed the passphrase while running 11.2 per SPo's suggestion, but I have to use the saved geli.key to unlock under 11.3.
I'm reluctant to export or detach and reimport since the documentation says, in bold in a warning block, "An encrypted pool cannot be reimported without a passphrase!".
Before removing the passphrase, there wasn't even a way to try to unlock it under 11.3: the web interface seemed broken, couldn't expand "Storage" or some of the other left menu sections and nothing happened when clicking anywhere in the "Pool" box in the dashboard view.
I am running 11.3 with encrypted pools without passphrase. Some of my pool had a passphrase and I removed them, I think.
To be sure, you could try shutting down your system and unplug your boot drive. Install a brand new boot drive with 11.3 already installed, and see if you can import your encrypted pool.
 

troun

Dabbler
Joined
Jul 13, 2013
Messages
33
@rio thanks, I attached your debug in ticket. I hope it helps, but it seems a tricky bug.
Did you create your pool in 9.X or 11.X?
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
@rio @faust7th would any of you have your 11.2 database saved by any chance ? I am unfortunately not able to reproduce this and am wondering what possibly differs in your environment and mine. Looking forward to hearing from you. Thank you
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
( Please don't post it here if you do, you can email them to me at waqar@ixsystems.com or a jira ticket with them attached would be nice as well )
Can you please provide output of "ls -al /data/geli" as well if you can in the email ? Thank you
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
@SPo @rio @faust7th can you please confirm if you had a whitespace character in your passphrase at the start or at the end ?
Also @SPo sorry I missed tagging you in my last reply, can you please confirm if you would have a config saved of your 11.2 BE before you upgraded ? If you do, can you please email it to me at waqar@ixsystems.com ? Thank you
 
Top