Pool won't decrypt after upgrade

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
I just upgraded my system to a Ryzen 5 3600 and I'm trying to get my Pools to decrypt. I also have a backup of the config (with passwords saved) from the working install.

I remember the passphrase, and I have the Geli.key. When I initially tried to decrypt I got a Middleware error.

I then detached the pool, and have been attempting to re-import which gives me this error.

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
await self.future
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 388, in __run_body
rv = await self.middleware.run_in_thread(self.method, *([self] + args))
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1066, in run_in_thread
return await self.loop.run_in_executor(self.__io_threadpool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 953, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/disk.py", line 652, in decrypt
raise CallError(f'The following devices failed to attach: {", ".join(failed)}')
middlewared.service_exception.CallError: [EFAULT] The following devices failed to attach: gptid/277201c8-7b95-11e7-9fa6-002590397f22, gptid/ed604efe-c574-11e7-85ba-002590397f22, gptid/503f85e9-7bc6-11e7-9fa6-002590397f22, gptid/eff6c209-7ba7-11e7-9fa6-002590397f22, gptid/0c2713c9-7b83-11e7-9fa6-002590397f22, gptid/eacb3826-39b2-11e7-8547-002590037356, gptid/e7f66d0f-39b2-11e7-8547-002590037356, gptid/d9bcc41c-39b2-11e7-8547-002590037356, gptid/6534706d-117d-11ea-adff-002590397f22, gptid/4d588b3f-4019-11e8-bdcf-002590397f22, gptid/23596f22-ef7a-11e8-b950-002590397f22, gptid/b401009e-7bb6-11e7-9fa6-002590397f22, gptid/d6766202-39b2-11e7-8547-002590037356, gptid/df80e633-39b2-11e7-8547-002590037356, gptid/e267dc68-39b2-11e7-8547-002590037356, gptid/e530014f-39b2-11e7-8547-002590037356

Geom Disk List gives me this: (with 15 others all looking fine)
Geom name: da0
Providers:
1. Name: da0
Mediasize: 3000592982016 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: ATA Hitachi HUS72403
lunid: 5000cca22cc7f466
ident: P8GKHGRP
rotationrate: 7200
fwsectors: 63
fwheads: 255

geli attach -k geli.key /dev/da1 gives me:
geli: Cannot read metadata from /dev/da5: Invalid argument.
 

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
I can't see a way to edit, but I initially started in 11.2 - Stable and after being unable to get it to work I upgraded to 11.3-BETA which didn't fix anything (though looks very nice)
 

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
Update: geli attach -k geli2.key /dev/da5p2
Gives me: geli: Wrong key for da1p2.
 
Joined
Oct 18, 2018
Messages
969
Hi @Dekkars

Quickly, for sanity do you mind surrounding the code way above with code tags similar to the following?

[code]
some code
[/code]


I just upgraded my system to a Ryzen 5 3600 and I'm trying to get my Pools to decrypt. I also have a backup of the config (with passwords saved) from the working install.
The config backup does not contain the decryption keys, unfortunately.

I remember the passphrase, and I have the Geli.key. When I initially tried to decrypt I got a Middleware error.
By what means did you try this? Did you have to try to import the pool and it gave you this error? Or perhaps, after the upgrade you tried to unlock the pool via the GUI and got this error?

Update: geli attach -k geli2.key /dev/da5p2
Gives me: geli: Wrong key for da1p2.
This is the wrong command. Your encrypted pools use devices in /dev/gptid. You can find out which devices by running the following command. sqlite3 /data/freenas-v1.db 'select * from storage_encrypteddisk;'. You'll see the correct paths there prepended with /gptid.

What I would do is first, test each drive manually with the key and passphrase you have via geli attach -k geli2.key /dev/gptid/<device>. If each and every drive in your pool works, then lock each drive with geli detach /dev/gptid/<device> and do the following.

Compare what your system thinks is the key with the key you have in hand via diff /data/geli/<key>.key geli2.key. You can identify the correct key file, if there are multiple, with sqlite3 /data/freenas-v1.db 'select vol_encryptkey from storage_volume;'. If you were able to unlock every drive by hand then I would expect these two files to be different. If they are, continue on, else go ahead and read below for three commands that would be useful to me in debugging. If they do match the solution may be as simple as cp /data/geli/<key>.key /data/geli/<key>.key_bak; cp geli2.key /data/geli/<key>.key, attempt to unlock the pool via the GUI (make sure you locked them all again previously), and if that works reboot and try to unlock the pool again. If that works you can safely get rid of the _bak key.

If the above does work you will also likely want to add and store a recovery key as well. Click the lock on your pool and select "Add Recovery Key" and download that key. Keep geli2.key and recovery.key and guard them well. The latter key does NOT use your passphrase and is intended as a last-ditch recovery key; you should keep it somewhere very safe as anyone with that key can unlock your drives without your passphrase.

If the above does not work could you report the full output of the following three commands?
Code:
$ sqlite3 /data/freenas-v1.db 'select * from storage_volume;'
$ sqlite3 /data/freenas-v1.db 'select * from storage_encrypteddisk;'
$ diff /data/geli/<key>.geli geli2.key
 
Joined
Oct 18, 2018
Messages
969
Ah, I also just caught that you're on 11.3 beta. I'm not there yet, only running 11.2-U6 and 11.2-U7. 11.3 may be different in ways I'm not familiar with. The commands above should not be destructive in any case though. FWIW I don't think your issue has to do with 11.2. In the future, you may consider posting to the forums before resorting to a beta release.
 

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
Ah, I also just caught that you're on 11.3 beta. I'm not there yet, only running 11.2-U6 and 11.2-U7. 11.3 may be different in ways I'm not familiar with. The commands above should not be destructive in any case though. FWIW I don't think your issue has to do with 11.2. In the future, you may consider posting to the forums before resorting to a beta release.

Valid. I think I may be screwed. I went back to my original hardware and re-uploaded the config. My passphrase won't unlock, which I'm guessing means I need the Geli key to make it work. The Geli key I thought worked isn't functioning (though I can't recall re-keying ever)

This is also compounded by the fact that I recently replaced the disk, which I'm now remembering re-keys everything.

I think this is just me, not an error (though I was hoping it was).

Wish me luck.
 
Last edited:
Joined
Oct 18, 2018
Messages
969
This is also compounded by the fact that I recently replaced the disk, which I'm now remembering re-keys everything.
There is misleading information in the User Guide about this.

Valid. I think I may be screwed. I went back to my original hardware and re-uploaded the config. My passphrase won't unlock, which I'm guessing means I need the Geli key to make it work. The Geli key I thought worked isn't functioning (though I can't recall re-keying ever)
Okay, before you do too many things and assume you're screwed, maybe it is worth taking a stab at resolving it?

First, gather all keys you possibly have. I know you have one currently named geli.key. I'm not sure what your situation is like exactly so it is hard to give exact advice. In general though, you're going to want to scrounge ALL keys you can.

Look in the /data/geli and download every key you see there.

In my prior post I suggested a few commands to run; did you run them? If so could you please post the output here? In addition, what is the output of ls /dev/gptid

Finally, you should attempt to unlock each of your devices with each key.
Code:
geli attach -k <key> /dev/gptid/277201c8-7b95-11e7-9fa6-002590397f22
geli attach -k <key> /dev/gptid/ed604efe-c574-11e7-85ba-002590397f22
geli attach -k <key> /dev/gptid/503f85e9-7bc6-11e7-9fa6-002590397f22
geli attach -k <key> /dev/gptid/eff6c209-7ba7-11e7-9fa6-002590397f22
geli attach -k <key> /dev/gptid/0c2713c9-7b83-11e7-9fa6-002590397f22
geli attach -k <key> /dev/gptid/eacb3826-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/e7f66d0f-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/d9bcc41c-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/6534706d-117d-11ea-adff-002590397f22
geli attach -k <key> /dev/gptid/4d588b3f-4019-11e8-bdcf-002590397f22
geli attach -k <key> /dev/gptid/23596f22-ef7a-11e8-b950-002590397f22
geli attach -k <key> /dev/gptid/b401009e-7bb6-11e7-9fa6-002590397f22
geli attach -k <key> /dev/gptid/d6766202-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/df80e633-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/e267dc68-39b2-11e7-8547-002590037356
geli attach -k <key> /dev/gptid/e530014f-39b2-11e7-8547-002590037356

The above assumes the devices have not changed their ids, which is likely. Please enter each of those commands one at a time for each key until it unlocks. Make a note of which key unlocked which device and report back any devices which did NOT unlock. For those which do not unlock at all it may be worth going through your key list again with that device and adding the -p[/code] flag prior tot he [icode]-k flag. The reason is that IF one of the keys you have is the so-called "recovery" key it doesn't use a passphrase and so you need to tell geli to not expect a passphrase.
 

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
Ok - I took a look at both keys and one is a 64 byte size key and the other is a 0 byte size key which I'm fairly sure means it is trash. However I ran both and tried all the disks and I got:

Code:
Geli: Wrong key for gptid/<DiskGptid>


ls /dev/gptid gives

Code:
0c2713c9-7b83-11e7-9fa6-002590397f22    b2c7c46d-7bb6-11e7-9fa6-002590397f22    e267dc68-39b2-11e7-8547-002590037356
224d28b3-ef7a-11e8-b950-002590397f22    b401009e-7bb6-11e7-9fa6-002590397f22    e42dcfa8-39b2-11e7-8547-002590037356
23596f22-ef7a-11e8-b950-002590397f22    d56f42ce-39b2-11e7-8547-002590037356    e530014f-39b2-11e7-8547-002590037356
277201c8-7b95-11e7-9fa6-002590397f22    d6766202-39b2-11e7-8547-002590037356    e7f66d0f-39b2-11e7-8547-002590037356
4d588b3f-4019-11e8-bdcf-002590397f22    d9bcc41c-39b2-11e7-8547-002590037356    eacb3826-39b2-11e7-8547-002590037356
503f85e9-7bc6-11e7-9fa6-002590397f22    de8143a3-39b2-11e7-8547-002590037356    ed604efe-c574-11e7-85ba-002590397f22
5752bf88-160f-11ea-9328-002590397f22    df80e633-39b2-11e7-8547-002590037356    eff6c209-7ba7-11e7-9fa6-002590397f22
6534706d-117d-11ea-adff-002590397f22    e158fac6-39b2-11e7-8547-002590037356



I don't think I'm necessarily screwed, but that my current keys definitely aren't any good (the one good key I have is from 2017). I accessed and worked on the server a lot from an old work computer that had full disk backups, and I've contacted my old IT head who is going to check for any .key files and get back to me.

Basically:

I was replacing the motherboard, processor, and RAM. In the course of the swap I had to reformat and reinstall freenas on the boot drive. I had a backup of what I THOUGHT was both keys, but it only contained one and it is really old.
 

Dekkars

Dabbler
Joined
May 18, 2017
Messages
13
WE'RE OK!

The original boot drive was a mirrored pair, and I assumed when I formatted and reinstalled it wiped both. However, in my ineptitude I only did one of them, so the other drive still had the original key still in the filesystem! Passphrase mounted all the disks!

I've downloaded new recovery keys and encryption key, and I'll try another swap on the processor and motherboard tomorrow.

Thank you for all your hard work, I really appreciate the help and attention you gave this.
 
Joined
Oct 18, 2018
Messages
969
I was replacing the motherboard, processor, and RAM. In the course of the swap I had to reformat and reinstall freenas on the boot drive. I had a backup of what I THOUGHT was both keys, but it only contained one and it is really old.
For the future you should take care to make absolutely certain you have the correct keys. If you don't maintain your keys you can lose access to your data and if you don't have backups or can't unlock them you lose all your data.

I've downloaded new recovery keys and encryption key, and I'll try another swap on the processor and motherboard tomorrow.
This is good news. Be sure you know that both of the keys you have work as expected before you do any swap.
 
Top