Reconnecting encrypted pools

SubnetMask

Contributor
Joined
Jul 27, 2017
Messages
129
I just finished converting one of my Supermicro 2U enclosures from a full system with motherboard, pros, controllers, NICs etc to a JBOD enclosure, so I now have two 2U JBOD enclosures that are connected to a R620 via the same controllers that I've been using with the Supermicro board. For part of this, I was also going to migrate to a HDD (or two) away from the USB stick I've been using, so prior to the change, I set up a fresh install of 11.1U7 on the new box, installed on the HDD, and at the time of the change, I made a fresh backup of the config and all of the keys.

After converting the first enclosure to be a JBOD and moving the controllers over to the R620, I booted the R620 and after it booted, I restored my config, after which of course it rebooted and did what it had to do to migrate my original config, and after booting after doing all of that, it detected all of the pools. Not surprisingly, they were all locked. Now, since I had imported my config and it saw all of the pools, one would think you would not have to import the pools, and that just using 'Unlock' would do the trick, and would prompt for a key if one didn't exist, but that's not what happened - it tried to unlock, couldn't and just said 'sorry'. So I tried to do an import, but this is where it got weird. It saw all disks when I looked at 'View Disks', but when I went to import, I told it to decrypt, supplied the key, but when it asked to select disks, there were only like five or six present in the list - One was a SATA disk that was part of a two disk stripe (It's a temporary location for backup data so if it were destroyed due to a disk failure, no big deal), the others were multiport SAS disks that are part of my other three pools. I didn't know what to do there, so I aborted without making any changes and ended up shutting down, pulling the new boot device and installing the original USB Key taht was runnin gwith the Supermicro board, booting that, and after that started a boot, did something related to the change of hardware, I assume and rebooted again, all pools came up unlocked and healthy, I just had to fix NIC relationships, but all was well after that. So for now, I'm running off that USB stick still.

Since then, I have destroyed the Backup pool to use the disks for something else for a period of time, and I also unmounted and re-mounted my DVR pool (To move the disks elsewhere without shutting the whole system down), which to re-mount, I imported the pool, at which point it asked for the key, asked me to select the disks, both of which were listed, and it did it's thing, re-mounted the pool and everything was there and happy.

So with the attempt to import after moving to the fresh install with my config restored, why were there only a few disks listed? Does it only list one disk from each pool when there are more than a certain number of pools and/or disks?

I did some reading and learned that the keys are located at /data/geli, so I looked there and found the keys, named what I assume are the GUIDs for the pools.

Since I restored the config, can I copy the geli keys from /data/geli on the USB key as is, and put them in the same location with the same names on the fresh install, after which on bootup, it'll find them and unlock the pools?
 

SubnetMask

Contributor
Joined
Jul 27, 2017
Messages
129
Nope, not a peep. But I did have a 'heart stopper' moment a few weeks ago, that in a way, deepened my question/confusion as to why I couldn't re-open my pools on a freshly installed and configuration restored instance.

So what happened was I was pulling empty caddies in preparation to install some new, freshly tested 8TB disks, and somehow, apparently (although I have not figured out HOW), one of the caddies contacted part of the backplane, apparently shorting something and shutting the whole enclosure off. My entire main data pool vanished in an instant. In the end, I first tried getting everything shut down and rebooting FreeNAS, after which the pool was still in not in a healthy state. In the end, I detached what it saw, went to import, told it to decrypt, supplied the key, and all of the proper disks were listed (This is what added to the confusion of trying to import the volumes after the fresh install), so I was able to select them and I was able to re-attach the pool and everything was present and came back online healthy. Slight segway, this is why I personally never spread disks for a pool, volume, etc across different enclosures - With all the disks in the same enclosure, when the whole thing vanished, the possibility of recovery existed, and in the end was possible (Essentially, it was only a little different from if the entire stack, controller and all, had powered off unexpectedly), but had there been two or more disks in the first enclosure, that pool would have been toast.

So anyway, like I said, after the config restore on the fresh install, when I tried an import, only a few disks were listed, which is where I stopped and reverted to the original USB key and brought everything back up. But logically, importing volumes makes total sense if you're talking about a fresh install where you don't have a backup... but when you restore the config, one would think that becasue you just restored the config, as such, it knows what disks are what and such, when you boot it up after restoring the config to the fresh install, you'd think you could select the 'unlock' option, and when the key wasn't present on the system, it prompts you for the key, you provide it and off you go. But what I got was essentially 'failed to unlock'. And the fact that there were a few disks listed when I tried to import doesn't fit with the possibility of having to detach the pools first like I had to do when the second enclosure powered down as there were no unassigned disks attached - every disk attached to the controller was assigned to a pool.

Like I said, if you do a fresh install and don't have a confg backup, having to import volumes makes total sense. But if you take a backup just before doing a clean install on some other form of media (like I did), then install a fresh copy and restore that config, because it knows everything about the config, including all disks and pools, except for the key to unlock them, you'd think that after doing a restore, the system would allow you to unlock the pool, prompting for the key when it wasn't found.

Is the config backup only useful for everything other than the pools (when they're encrypted)? So once a config is restored, you need to detach the pools it sees (but disconnected/offline becuase it doesn't have the keys to decrypt them), where you must detach them all and re-import them after a restore such as that? Or, as I originally asked, since I have access to the original, functioning boot media, can I copy the original geli keys exactly as named (which again, I assume are named with the GUID volume ID) from that media into the same path on the new media and they'll all mount properly on boot?
 

charlie89

Explorer
Joined
Dec 26, 2013
Messages
55
After restoring the config to another system, all encrypted pools must be detached and attached, otherwise decrypting the pool fails.
Remember that you need your pool encryption passphrase and encryption key to attach the pool.
I would advise you to check all pool related settings like shares, scrubs, smart checks, snapshots,... the last time i did this all snapshot tasks were gone and for over 2 months i didn't had any snapshot.
 
Top