Resilvered my main storage pool and system dataset opportunistically moved to it?

[Teeps]

Hello,

My issue is very similar to this resolved ticket : https://jira.ixsystems.com/browse/NAS-102437

I replaced a drive in an encrypted pool by following the instructions here:

9. Storage — FreeNAS®11.2-U7 User Guide Table of Contents

www.ixsystems.com

1. Replaced disk
2. Add Recovery Key
3. Encryption Rekey
4. Add passphrase

The bug that is mentioned in the link above was not addressed in that report, that the new web UI doesn't explain why the passphrase cannot be set, and it also doesn't address the more fundamental issue of why the system dataset suddenly blocks the main storage pool from getting a new passphrase. Where was the system dataset previously that it is now on my main storage pool? Why did it change to that pool once I did "Encryption Rekey"? Why do I have to go googling the system dataset when I started out with a simple drive replacement?

Should I open two new bugs?
* New web UI does not explain system dataset is blocking passphrase creation
* Resilvering encrypted pool can cause system dataset to migrate to that pool


A third minor gripe is that the documentation isn't very clear on if rekeying is necessary/recommended. It would be nice if there was an "(Optional)" or a second sentence explaining why one would want to rekey the pool after replacing a disk. This is to render the replaced disk forever unreadable for safe disposal/re-use, correct? I'm not sure if documentation suggestions also get bugs in Jira.

 

[gary_1]

iirc when you replace a disk, assuming you unlocked your pool with your main key (and not the recovery key). Then all drives in the pool will be unlockable with that same main key in the future, however the newly replaced drive will NOT be unlockable with your recovery key. For this reason you have to add a recovery key (or more to the point, generate a new recovery key for all drives), so all drives now have the same recovery key.

The instruction to do an encryption rekey to make a new main key, I believe (don't quote me though) is as you mention, to ensure the active disks now use a different key to the drive you'll be disposing of. I do not believe you _have_ to rekey though.

HOWEVER... if you unlocked your pool using a recovery key, the reverse situation will occur. Freenas will only have access to a key that's able to decrypt the recovery key slot, so your newly replaced disk will have a valid recovery key slot. All drives in the pool will thus be unlockable by your recovery key, but the new drive will not have a valid main key and it will not be possible to unlock with a main key. The encryption rekey in this case is vital.

So it may also be that the docs are just ensuring both keys are regenerated so that no matter which key was used to unlock the pool, all drives have a valid main and recovery key.

I think the docs are not clear on the reason nor what is or is not required, so people cannot reason as to what is safe to do or not do.

Also, I think it'd be good if freenas had an option during resilver to upload both your main key and recovery key so that it has both keys available and can encrypt the disk decryption key for both slots, ensuring the new drive gets two valid keys without the user having to change the keys used to unlock their pool. This would remove the need for any rekeying at all, which can be a bit of a pain as you have to ensure your "safe" storage location for the previous recovery key is updated, whether that's some other disk, paper print out in a safe or whatever.

If the above was possible, the docs could then recommend rekeying rather than providing both keys with reasons why, if there's a data security reason for disk disposal. Users can then have sufficient information to decide if that's a risk they're ok to take in order to continue to use the same main/recovery key for their pool.