Replace encrypted drive

[Ericloewe]

Now you're getting into a problem called "the GUI people don't seem to understand how GELI works", for which there is a bug ticket somewhere.

Both keys (passphrase+key 1 and key 2) should really be done in sequence automatically - download the new key, verify that it's correct, download the second key, verify.

What's happening? FreeNAS makes you do all this manually, which leads to user mistakes. This stuff isn't rocket science, but someone needs to follow the feature from end to end to make sure it makes sense.

 

[psoni]

So, here's what I saw in the doc (about waiting for resilvering)-

8.1.10.1. Replacing an Encrypted Drive
If the ZFS pool is encrypted, additional steps are needed when replacing a failed drive.
First, make sure that a passphrase has been set using the instructions in Encryption before attempting to replace the failed drive. Then, follow the steps 1 and 2 as described above. During step 3, a prompt will appear to input and confirm the passphrase for the pool. Enter this information then click the Replace Disk button. Wait until the resilvering is complete.

Next, restore the encryption keys to the pool. If the following additional steps are not performed before the next reboot, access to the pool might be permanently lost.

1. Highlight the pool that contains the recently replaced disk and click the Encryption Re-key button in the GUI. Entry of the root password will be required.
2. Highlight the pool that contains the disk you just replaced and click Create Passphrase and enter the new passphrase. The old passphrase can be reused if desired.
3. Highlight the pool that contains the recently replaced disk and click the Download Key button to save the new encryption key. Since the old key will no longer function, any old keys can be safely discarded.
4. Highlight the pool that contains the disk that was just replaced and click the Add Recovery Key button to save the new recovery key. The old recovery key will no longer function, so it can be safely discarded.

 

[Ericloewe]

The latest version of the docs no longer says that, which saves me from asking to have it fixed.

 

[IQless]

This might be a stupid question, but..
When the documentation states "Immediately restore the encryption keys to the pool.", do they mean to "Re-Key"? Or am I missing something here?

 

[Ericloewe]

Yeah, that's it.

 

[Ericloewe]

Adding sets up a key. Removing deletes the metadata that contains the encrypted master key, I guess.

Will Rekeying works when no passphrase is used?

I'm not sure what FreeNAS does, but possibly.

 

[pro lamer]

When the documentation states "Immediately restore the encryption keys to the pool.", do they mean to "Re-Key"? Or am I missing something here?

Yeah, that's it.

This part of the docs might be worth rephrasing then... I can't remember from the top of my head whom to @ mention... ;-)

Sent from my phone

 

[pro lamer]

re-key as soon

I think it is safe. If you don't then the replaced disk will not be available

I must be missing something. Does that mean that the new disk is being given some other keys into first or second slot (other comparing to the disks already in pool) ? Or maybe no keys into the two slots at all (just kept in RAM) ? And the new geli key has to be copied into the other disks? (Or uploaded into all the disks, including the new one? ).

It seems opposite to what I could remember until now - that rekeying could be skipped. There's even some older thread regarding this matter:

. When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys.

That said, it's probably not a strict requirement that new user keys be used, but it makes some sense to avoid leaking information to attackers. FreeNAS makes it a requirement.

Sent from my phone

 

[Ericloewe]

Does that mean that the new disk is being given some other keys into first or second slot (other comparing to the disks already in pool) ?

Yes, which is why it's necessary to re-key the disks.

It seems opposite to what I could remember until now - that rekeying could be skipped. There's even some older thread regarding this matter:

If you skip it, you need to manually keep track of the keys for each disk.

 

[pro lamer]

Yes, which is why it's necessary to re-key the disks.

If you skip it, you need to manually keep track of the keys for each disk.

Is it something new? Has it been introduced "recently"? As an opposite to only putting the old disks keys at risk like this:

You're leaking an entire disk's worth of data encrypted with a key that sits in a known location, encrypted. It's not likely at all that it's going to be decrypted in useful time, but if it is, you can use it on the other old disks

but did the new key disk use to be given the same key in the past?

Sent from my phone

 

[SMnasMAN]

Hello, Gen8 Runner. Not too long ago I made a minor change to the documentation dealing specifically with your issue. The change says to restore the encryption keys immediately after clicking REPLACE DISK instead of waiting on the resiliver to complete. This is a much safer process.

This change can be found at https://github.com/freenas/freenas-docs/pull/549.

It looks like the manual STILL has the old info (which i agree with the OP here, makes this a dangerous and CONFUSING process. one which is critical to avoid data loss).

everyone will scream- read the manual, go to the manual. but unfortunately there are many times (ive found) that the manual is confusing and lacking detail (and at times consistency).

this is from the 11.2u5 (current manual):

During step 3, there will be a prompt to enter and confirm the passphrase for the pool. Enter this information, then click REPLACE DISK.

Wait until re-silvering is complete before restoring the encryption keys to the pool. Restore the encryption keys before the next reboot or access to the pool will be permanently lost.
(no big deal, right?)

so again, @aaron.stjohn , do we wait for re-silver, or do we restore encryption key " immediately after clicking REPLACE DISK instead of waiting on the resiliver to complete" (which i agree is much safer).

Im in this scenario right now, with a disk that is acting up, that is part of a enc pool. I have nearly 6 pages of notes ive made and practiced on how to replace disks, but still cant get a straight answer/process (official) on replace disk on enc pool..

maybe ixsys coudl add an actual step by step example to the manual, instead of a single paragraph on this critical process. Thanks

 

[Apollo]

Wait until re-silvering is complete before restoring the encryption keys to the pool.

Restore the encryption keys before the next reboot or access to the pool will be permanently lost.

@SMnasMAN, Technically, you don't restore the key as it implies you are expecting to upload the key to the array which isn't true.
To me having to wait for the resilvering to complete isn't very productive and if you wait for it to complete, you will have to remember not to reboot before updating the key. Worst case scenario, you will reboot and upon reboot, the newly added disk will not attach to the array but the array will not be made unavailable. You will have to rebuilt it again by replacing the drive and attaching it again.

So I think it makes sense to update the key once resilvering has started.
But then again, you still want to have backups just to be on the safe side.

 

[SMnasMAN]

thanks for the reply / info. Im about to try all this and will update, im depending mainly on forum posts as im really disappointed in the detail provided in the manual (mainly the "steps", which is the most important). Its unfortunate that lines like this are thrown around in the manual "access to the pool will be permanently lost. " but still no clear , detailed info/steps are provided.

 

[pro lamer]

access to the pool will be permanently lost.

IIRC only the new disk would be unlocked and all the other would stay locked. It's from the top of my head so put a grain of salt...

Sent from my phone

 

[aaron.stjohn]

It looks like the manual STILL has the old info (which i agree with the OP here, makes this a dangerous and CONFUSING process. one which is critical to avoid data loss).

everyone will scream- read the manual, go to the manual. but unfortunately there are many times (I've found) that the manual is confusing and lacking detail (and at times consistency).

this is from the 11.2u5 (current manual):

During step 3, there will be a prompt to enter and confirm the passphrase for the pool. Enter this information, then click REPLACE DISK.

Wait until re-silvering is complete before restoring the encryption keys to the pool. Restore the encryption keys before the next reboot or access to the pool will be permanently lost.
(no big deal, right?)

so again, @aaron.stjohn , do we wait for re-silver, or do we restore encryption key " immediately after clicking REPLACE DISK instead of waiting on the resiliver to complete" (which i agree is much safer).

Im in this scenario right now, with a disk that is acting up, that is part of a enc pool. I have nearly 6 pages of notes I've made and practiced on how to replace disks, but still can't get a straight answer/process (official) on replace disk on enc pool..

maybe ixsys coudl add an actual step by step example to the manual, instead of a single paragraph on this critical process. Thanks

Hello, SMnasMAN.

I apologize for the confusion. To answer your question, you should restore the encryption key immediately after replacing an encrypted disk. Restoring the encryption key on the new disk does not affect the resilvering process.

Regarding the current docs, I have found where my original change to the docs was reverted. This was a mistake and I'm currently fixing it to say that the encryption key should immediately be restored after replacing the disk.

Thank you for pointing this out.

EDIT: I've done a small rewrite of the process for replacing an encrypted disk. I hope this helps: https://github.com/freenas/freenas-docs/pull/1497/files

 

[emptyBox]

Hello, SMnasMAN.
I apologize for the confusion. To answer your question, you should restore the encryption key immediately after replacing an encrypted disk. Restoring the encryption key on the new disk does not affect the resilvering process.
Regarding the current docs, I have found where my original change to the docs was reverted. This was a mistake and I'm currently fixing it to say that the encryption key should immediately be restored after replacing the disk.
Thank you for pointing this out.
EDIT: I've done a small rewrite of the process for replacing an encrypted disk. I hope this helps: https://github.com/freenas/freenas-docs/pull/1497/files

 

[emptyBox]

Following the directions of aaron.stjohn that he loaded onto github here, https://github.com/freenas/freenas-docs/pull/1497/files
I add the recovery key, then rekey the pool. However, when I go to set a new passphrase, I get "error creating passphrase for pool tank1"..

I do have all the keys downloaded. Should I begin a backup in case I will lose this pool?

 

[aaron.stjohn]

@emptyBox It never hurts to back up your data. I would recommend this.

However, the problem here sounds like a bug. Please open a bug ticket on our ticket platform with at least this information:

1. The version of FreeNAS the problem was seen on.
2. Attach a dubug. (Go to System > Advanced > Save Debug)
3. A clear, concise summary of the issue and how to reproduce it.

Thank you!

 

[emptyBox]

So is it likely that I can lose my data? The drive seems to have resilvered. I haven't rebooted the machine yet because the prospect of being unable to unlock my pool is something I don't particularly want to deal with.

Is this typical? Should I be concerned with being unable to decrypt the pool? I did add a new key and rekey it, as the instructions state in your git.

I'm currently updated to the latest version, although, I think an update came within the past few weeks or so that I might have missed, but I am on the stable current version, I guess one update behind.

 

[PhiloEpisteme]

I wanted to chime in here with concrete examples of what happens on an 11.2-U6 machine when you resilver a pool both with and without a passphrase.

First, I set up a pool which did not have a passphrase and resilvered one of the disks. During all of this I watched /var/log/debug.log.

Code:

Oct 18 20:24:40 nas uwsgi: [middleware.notifier:188] geli init -s 4096 -l 256 -B none -P -K /data/geli/4cbcca78-31fb-4f41-81fd-86522f484c3c.key gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630 -> 0
Oct 18 20:24:40 nas uwsgi: [middleware.notifier:188] geli attach -p -k /data/geli/4cbcca78-31fb-4f41-81fd-86522f484c3c.key gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630 -> 0

The above logs suggest that during resilver the new drive is given the same User Key 0 as the original pool. This is reasonable given that FreeNAS has access to that key since it lives on the system in /data/geli. To keep testing I did the following:

• Trick the system into thinking that the pool can be locked. sqlite3 /data/freenas-v1.db 'UPDATE storage_volume SET vol_encrypt = 2 WHERE vol_name = "test";
  
  
• Lock the pool in the UI.
  
  
• Test both keys against both disks by trying to unlock and then lock them. geli attach -P -K 4cbcca78-31fb-4f41-81fd-86522f484c3c.key /dev/gptid/{device}. geli detach /dev/gptid/{device}. geli attach -P -K recovery.key /dev/gptid/{device}. geli detach /dev/gptid/{device}.
  
  
• As expected, the disk we never replaced unlocks with both keys.
  Code:

  $ geli attach -k ~/4cbcca78-31fb-4f41-81fd-86522f484c3c.key -p /dev/gptid/a88c7e3c-f1e4-11e9-86e5-00074334f630
  $ geli detach /dev/gptid/a88c7e3c-f1e4-11e9-86e5-00074334f630
  $ geli attach -k ~/recovery.key -p /dev/gptid/a88c7e3c-f1e4-11e9-86e5-00074334f630
  $ geli detach /dev/gptid/a88c7e3c-f1e4-11e9-86e5-00074334f630
  $ geli attach -k ~/4cbcca78-31fb-4f41-81fd-86522f484c3c.key -p /dev/gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630
  $ geli detach /dev/gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630
  $ geli attach -k ~/recovery.key -p /dev/gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630
  geli: Wrong key for gptid/4cbe06c9-f1e5-11e9-86e5-00074334f630.
  

You see that both keys worked for the original disk and User Key 0 worked for the replacement disk; only the recovery key was not valid for the replacement disk.

I did the same thing with a pool which had a passphrase set and got the same results. From the logs

Code:

Oct 20 21:40:09 baret uwsgi: [middleware.notifier:179] Popen()ing: geli init -s 4096 -l 256 -B none -J /tmp/tmpnhzde3qa -K /data/geli/4cbcca78-31fb-4f41-81fd-86522f484c3c.key gptid/2d1070a7-f382-11e9-86e5-00074334f630
Oct 20 21:40:17 baret uwsgi: [middleware.notifier:188] geli init -s 4096 -l 256 -B none -J /tmp/tmpnhzde3qa -K /data/geli/4cbcca78-31fb-4f41-81fd-86522f484c3c.key gptid/2d1070a7-f382-11e9-86e5-00074334f630 -> 0

Note the -J flag above setting the passphrase of the drive.

After resilver and before rekeying/adding the passphrase back both User Key 0 w/passphrase and User Key 1 worked on the original drives and only User Key 0 w/passphrase worked on the replacement drive.

@emptyBox if you have kept copies of ALL keys it is likely you can get this sorted out. Is the pool currently unlocked? And do you have copies of the primary and recovery keys you expect to work? If so, give me the output of zpool status <pool> so I can see the drive identifiers and I can give you the commands to run via CLI to get your keys guaranteed set correctly.