Volte
Dabbler
- Joined
- Feb 11, 2016
- Messages
- 19
Hi all!
I know what you're thinking: Oh no, another "my data is encrypted and I can't recover it!". I understand the risks that come with running encrypted disks. I have a peculiar situation none-the-less and would love a few additional brains on the matter.
(I figured I'd take my long narrative and make it hide/showable. If that is not helpful let me know and I'll change it)
TL;DR:
Very best,
~ Dale
I know what you're thinking: Oh no, another "my data is encrypted and I can't recover it!". I understand the risks that come with running encrypted disks. I have a peculiar situation none-the-less and would love a few additional brains on the matter.
(I figured I'd take my long narrative and make it hide/showable. If that is not helpful let me know and I'll change it)
I've got four drives (/dev/ada0 - /dev/ada3) set up as mirror/striped. I run them encrypted, and it's been pretty stable since FreeNAS 9.x
I've had a couple of close calls before where I've nearly lost all my data because somehow the recovery key I had saved was not working. (I've been lurking on the forums for a while now, and done many searches over the years around this stuff and it seems there is some sort of known issue with the downloaded recovery key, etc, but that may not be particularly relevant right now, because the situation is what it is).
Anyhow, I avoided disaster by scouring all my backups and computers and I found a rogue geli key laying around that ended up working. Hooray! This was a few years ago.
Fast forward to today, and I'd been getting daily reports about my boot volume being in a degraded state. (booo). So, aside from waiting two weeks to find time to address it, I was a relatively good citizen and got myself some new USB Drives to move over to. (This is not the first time, in fact, the first close call I mentioned above was similar in that the boot drive would not... boot). Anyhow, I was running 11.2-U4-1 and figured it would be a good time to upgrade anyway. I have daily backups of my `freenas-v1.db`, so I wasn't worried, but I downloaded a copy just in case. I grabbed a fresh 11.2-U6 threw it on one new USB drive, and installed onto a second new USB drive.
Meanwhile (and this was stupid, but spoiler alert it still boots) I took my degraded 11.4-U4-1 USB drive and ran some diagnostics on it, including block recovery (yes, data corruption can occur). So the fresh install finishes, and I reboot into it. I upload my backedup settings, and it reboots a few times. Once it comes back up, I mosey on over to the Storage section, I see my pool sitting there, locked, as expected. I click the unlock button, add my password, upload the recovery key (which I'd quadruple checked during and after my last close-call, and saved it in a very safe place, and wrote down the MD5 hash of it, and saved that as well, just to be sure).
Well, what the heck!? So I try it a few more times, I try it from the command line (
At this point, the cold sweats start to appear. Ok, so the recovery key that I quadruple checked on new clean installs prior to storing away for a rainy day, doesn't in fact work. I was away from my server at this point, but I brought the thumb drive with me. I tried booting from it on a VirtualBox with the intent on checking for and recovering a different geli key (maybe I goofed again and saved the wrong one?), but to no avail, so I was sure at this point I'd corrupted the drive to a point where it couldn't boot. So, I began a data recovery process on it for 3 days in hopes of maybe finding the right geli recovery key. That proved fruitless, but I didn't want to stop the recovery process until it was done.
Once it'd finished, and I couldn't find the key, I said screw it, let me try booting from it (now that I was back with the server). Sure enough, it booted up (although, it began the upgrade process to 11.2-U6, which I stopped halfway. It booted up, and I started poking around. I unlocked the pool with just the password, as usual (since I assume it's using the geli key in the /data/geli/ directory. Interestingly though, the /data/geli/[hash].key geli key was at zero bytes. I thought that was odd after I'd scp'd it to my local computer. Well clearly that's useless (but how did it unlock the drives anyway? ). So I logged into the GUI, and started clicking around. I saw a "encryption rekey" and I figured what the heck, might as well try this while I still have access to my encrypted pool!. So I did that, and it wrote over the zero byte geli key, with 64 bytes. That looks better! So I downloaded that (both through the GUI, and through SCP to be sure), and now confident that I was armed with the correct key, I shutdown, swapped the USB thumb drives, and booted into the fresh 11.2-U6 install. I headed over to the storage section, went to unlock my pool, and again, the same error (as above). I also tried (again) to do it from the command line.
So, here I am.
I've had a couple of close calls before where I've nearly lost all my data because somehow the recovery key I had saved was not working. (I've been lurking on the forums for a while now, and done many searches over the years around this stuff and it seems there is some sort of known issue with the downloaded recovery key, etc, but that may not be particularly relevant right now, because the situation is what it is).
Anyhow, I avoided disaster by scouring all my backups and computers and I found a rogue geli key laying around that ended up working. Hooray! This was a few years ago.
Fast forward to today, and I'd been getting daily reports about my boot volume being in a degraded state. (booo). So, aside from waiting two weeks to find time to address it, I was a relatively good citizen and got myself some new USB Drives to move over to. (This is not the first time, in fact, the first close call I mentioned above was similar in that the boot drive would not... boot). Anyhow, I was running 11.2-U4-1 and figured it would be a good time to upgrade anyway. I have daily backups of my `freenas-v1.db`, so I wasn't worried, but I downloaded a copy just in case. I grabbed a fresh 11.2-U6 threw it on one new USB drive, and installed onto a second new USB drive.
Meanwhile (and this was stupid, but spoiler alert it still boots) I took my degraded 11.4-U4-1 USB drive and ran some diagnostics on it, including block recovery (yes, data corruption can occur). So the fresh install finishes, and I reboot into it. I upload my backedup settings, and it reboots a few times. Once it comes back up, I mosey on over to the Storage section, I see my pool sitting there, locked, as expected. I click the unlock button, add my password, upload the recovery key (which I'd quadruple checked during and after my last close-call, and saved it in a very safe place, and wrote down the MD5 hash of it, and saved that as well, just to be sure).
Code:
Error: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 219, in wrapper response = callback(request, *args, **kwargs) File "./freenasUI/api/resources.py", line 949, in unlock form.done(obj) File "./freenasUI/storage/forms.py", line 2847, in done raise MiddlewareError(msg) freenasUI.middleware.exceptions.MiddlewareError: [MiddlewareError: Volume could not be imported: 4 devices failed to decrypt]
Well, what the heck!? So I try it a few more times, I try it from the command line (
geli attach -k [geli_key_file] [dev_to_unlock]
), which informs me that geli: Wrong key for ada3p2
etc.At this point, the cold sweats start to appear. Ok, so the recovery key that I quadruple checked on new clean installs prior to storing away for a rainy day, doesn't in fact work. I was away from my server at this point, but I brought the thumb drive with me. I tried booting from it on a VirtualBox with the intent on checking for and recovering a different geli key (maybe I goofed again and saved the wrong one?), but to no avail, so I was sure at this point I'd corrupted the drive to a point where it couldn't boot. So, I began a data recovery process on it for 3 days in hopes of maybe finding the right geli recovery key. That proved fruitless, but I didn't want to stop the recovery process until it was done.
Once it'd finished, and I couldn't find the key, I said screw it, let me try booting from it (now that I was back with the server). Sure enough, it booted up (although, it began the upgrade process to 11.2-U6, which I stopped halfway. It booted up, and I started poking around. I unlocked the pool with just the password, as usual (since I assume it's using the geli key in the /data/geli/ directory. Interestingly though, the /data/geli/[hash].key geli key was at zero bytes. I thought that was odd after I'd scp'd it to my local computer. Well clearly that's useless (but how did it unlock the drives anyway? ). So I logged into the GUI, and started clicking around. I saw a "encryption rekey" and I figured what the heck, might as well try this while I still have access to my encrypted pool!. So I did that, and it wrote over the zero byte geli key, with 64 bytes. That looks better! So I downloaded that (both through the GUI, and through SCP to be sure), and now confident that I was armed with the correct key, I shutdown, swapped the USB thumb drives, and booted into the fresh 11.2-U6 install. I headed over to the storage section, went to unlock my pool, and again, the same error (as above). I also tried (again) to do it from the command line.
So, here I am.
TL;DR:
- Degraded boot drive (USB) 11.2-U4-1
- Run encrypted pool (four drives, mirrored/striped)
- Have quadruple confirmed working (apparently not) geli_recovery.key
- Fresh install on new USB (11.2-U6).
- Try to unlock pool with passphrase and recovery key, fails with error (above)
- Try finding a different geli_recovery.key.
- Reboot into degraded (but working) 11.2-U4-1 drive, see that geli key is 0bytes.
- Rekey pool. Download geli_recovery.key (note: different MD5 from "original" one).
- New install of 11.2-U6 still doesn't accept key and passphrase.
- Have tried the command line as well.
Very best,
~ Dale