How to remove encryption from a ZFS volume (while keeping the data)

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Hi, all,

my first post - I have a little procedure to share. I successfully removed the geli encryption from a live ZFS pool in FreeNAS 9.1.1 with the following steps:

0. Make sure you either have a separate backup of your data, or are willing to take the risk of losing everything. Second, please do not blindly follow these instructions if you do not know what you are doing. Third, the procedure works on one disk at a time. So if you run a RAIDZ2, you should be sufficiently safe. I would not dare to do this with data I need on a RAID1 or mirror. Nuff said ... let's start.

1. Scrub your pool to make sure all disks are in good condition.

2. Get the IDs of your zpool's devices:

[root@freenas-pmh] ~# zpool status
pool: zfs
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b4a21304-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
gptid/b4f40dbd-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
gptid/b5527faa-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0

errors: No known data errors

3. Start with the first disk - take offline and remove geli:

[root@freenas-pmh] ~# zpool offline zfs gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32.eli
[root@freenas-pmh] ~# geli detach gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32.eli
[root@freenas-pmh] ~# zpool status
pool: zfs
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
config:
NAME STATE READ WRITE CKSUM
zfs DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/b4a21304-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
gptid/b4f40dbd-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
gptid/b5527faa-8ec4-11e2-a224-28924a2bff32.eli ONLINE 0 0 0
5939868321408276145 OFFLINE 0 0 0 was /dev/gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32.eli
errors: No known data errors

4. Replace formerly encrypted device with unencrypted one:

[root@freenas-pmh] ~# zpool replace zfs 5939868321408276145 gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32
[root@freenas-pmh] ~# zpool status
...
replacing-3 OFFLINE 0 0 0
5939868321408276145 OFFLINE 0 0 0 was /dev/gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32.eli
gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32 ONLINE 0 0 0 (resilvering)

5. Remove information about encryption from FreeNAS' config database

I'm not quite sure if this is strictly necessary. I did it, and it definitely did not hurt. FreeNAS keeps track of which devices are encrypted. So I wanted to make sure it treats the disks correctly.

[root@freenas-pmh] ~# /usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32';"

6. Wait for the resilvering to finish

7. Repeat 2. - 6. for each remaining disk. In the end your pool should look like this:

[root@freenas-pmh] ~# zpool status
pool: zfs
state: ONLINE
...
config:
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b4a21304-8ec4-11e2-a224-28924a2bff32 ONLINE 0 0 0
gptid/b4f40dbd-8ec4-11e2-a224-28924a2bff32 ONLINE 0 0 0
gptid/b5527faa-8ec4-11e2-a224-28924a2bff32 ONLINE 0 0 0
gptid/b5ae2aed-8ec4-11e2-a224-28924a2bff32 ONLINE 0 0 0
errors: No known data errors

Note the absence of ".eli" from the device IDs. Check, if there are any entries left in the config database for encrypted disks:

[root@freenas-pmh] ~# /usr/local/bin/sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"

8. Stop all sharing services depending on the ZFS volume

9. Detach volume from the GUI - double check not to mark disks as new (i.e. erase them)

10. Reboot

11. Auto-import volume, restart sharing services

12. Done

Kind regards
Patrick
 

EEE3

Dabbler
Joined
Nov 27, 2013
Messages
12
I just signed up for an account to say thank you for taking the time to document and share this. I had been thinking about removing the encryption on my pool using this exact approach but figured I must be missing something because no one else mentioned doing it... glad to hear you were successful.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
This makes me wonder if there's a way to do this to encrypt a pool.. hmm....
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
It should work, but it is a slightly bigger challenge as you are creating things (encryption key file in /data/geli, geli provider, config DB records, ...) instead of deleting. There is also one encryption related table the original post omits. The storage_volume table has a column vol_encryptkey that stores the name of the /data/geli key file and a vol_encrypt column indicating that the volume is encrypted (0: not encrypted, 1: encrypted with a key file, 2: encrypted with a key file and a passphrase). I'm actually a bit surprised that the steps above work without zeroing the vol_encrypt flag at the end.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
To answer Dusan,

I'm pretty sure that it is precisely the export/detach and re-import of the pool that takes care of that. That's why I did it in the first place ;) And that's why I said I do not know if deleting the entries from storage_encrypteddisk is really necessary. Possibly they are just deleted when detaching, too. But since I do not know ...

As you can see I do not have that flag set, anymore:

[root@freenas-pmh] ~# /usr/local/bin/sqlite3 /data/freenas-v1.db 'select * from storage_volume;'
14043918962020988798|zfs||ZFS|0|1

The encryption flag is the second last column.

Kind regards,
Patrick
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
You are right, I should have realized that. Export/detach removes the record from storage_volume and import recreates it.
 

balrog76

Dabbler
Joined
Jul 25, 2015
Messages
10
I hope someone can help. I followed @Patrick M. Hausen s great guide. But alas, my freenas server decided to boot some time after resilvering the second disk of five.

Good thing: As far as I know, there was no resilvering going on when the server booted.
Bad thing: I had only done the zpool offline disk/geli detach disk/zpool replace disk on 2 of the five disks. So Freenas was very confused when coming back up.

my progress before boot:
zpool offline mixed gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.eli
geli detach gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool replace mixed 16498696492389751542 gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b

zpool offline gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.eli
geli detach gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool replace mixed 12773575498546196593 gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b

freenas# /usr/local/bin/sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"
1|1|gptid/97fab267-c3af-11e3-b1f9-bc5ff4cad02b|
1|2|gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b|
1|3|gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b|
1|4|gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b|
1|5|gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b|

Is there any chance I can recover from this? Maybe unlock the remaining disks from commandline, and finich the unencrypt operation?

Any help is very appreciated.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
I'd try to manually attach the still encrypted eli volumes. Then ZFS should be able to import the pool and you could continue the process.

HTH
Patrick
 

balrog76

Dabbler
Joined
Jul 25, 2015
Messages
10
Hi @Patrick M. Hausen

Thank you for taking you time to anwer. Worked like a charm.

For anyone else stumbling into this problem. Here is how to do it.

One small addition. I tried to unlock and mount the volume after the reboot. At this point Freenas still expected all 5 disks to be encrypted, and thus made geli attach to all of them. Including the two I had already unencrypted. It then proceded to puke and roll over. Which was where I was at when I wrote Patrick for help.

To get out of that I started the process with detaching the two unencrypted disks. I did not have to manually attach the remaining 3, as the GUI had already done that when it tried to import them all. So I was ready. Here is how it went:


log from when the GUI tried to attach all 5. See how the last two fails because they are unencrypted.
Dec 5 00:14:29 freenas GEOM_ELI: Device gptid/97fab267-c3af-11e3-b1f9-bc5ff4cad02b.eli created.
Dec 5 00:14:29 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 5 00:14:29 freenas GEOM_ELI: Crypto: hardware
Dec 5 00:14:31 freenas GEOM_ELI: Device gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b.eli created.
Dec 5 00:14:31 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 5 00:14:31 freenas GEOM_ELI: Crypto: hardware
Dec 5 00:14:32 freenas GEOM_ELI: Device gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b.eli created.
Dec 5 00:14:32 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 5 00:14:32 freenas GEOM_ELI: Crypto: hardware
Dec 5 00:14:34 freenas GEOM_ELI: Device gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.eli created.
Dec 5 00:14:34 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 5 00:14:34 freenas GEOM_ELI: Crypto: hardware
Dec 5 00:14:35 freenas GEOM_ELI: Device gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.eli created.
Dec 5 00:14:35 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 5 00:14:35 freenas GEOM_ELI: Crypto: hardware
Dec 5 00:14:36 freenas ZFS WARNING: Unable to attach to gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.
Dec 5 00:14:36 freenas ZFS WARNING: Unable to attach to da2p2.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to da1p2.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to da2p2.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.
Dec 5 00:14:37 freenas ZFS WARNING: Unable to attach to da1p2.




freenas still thinks all 5 are encrypted.
freenas# /usr/local/bin/sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"
1|1|gptid/97fab267-c3af-11e3-b1f9-bc5ff4cad02b|
1|2|gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b|
1|3|gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b|
1|4|gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b|
1|5|gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b|

startig with detaching the two unencrypted drives
here goes nothing
geli detach gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b.eli
geli detach gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool import mixed

In case of another unprovoked boot i decided to remove the "encrypted" flag for the two disks, from the freenas base right away.
/usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/d8f66f43-5dcf-11e3-aa6a-bc5ff4cad02b';"
/usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/d98d5e52-5dcf-11e3-aa6a-bc5ff4cad02b';"

I started a scrub after the reboot. It's now at 97% without errors. I am confident data is ok. So stopping it at this point.
zpool scub -s mixed
The gui reported no jails. So exported the volume and imported it with -R /mnt to get the right mountpoints. I would love to have my jails
available for the 6 days it will take to unencrypt the remaining 3 drives.
zpool export mixed
zpool import -R /mnt mixed
Well, that went fine, but still cant start jails. So I am rebooting at this point.
Reboot goes fine.
I can now import the pool from the gui, as freenas now knows that 2 of the drives are unencrypted. Works like a charm.

Now for the rest of the disks.

zpool offline mixed gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b.eli
geli detach gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool replace mixed 8884692931792616788 gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b
/usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b';"

zpool offline mixed gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b.eli
geli detach gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool replace mixed 16482857467227878109 gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b
/usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/dabf4250-5dcf-11e3-aa6a-bc5ff4cad02b';"

zpool offline mixed gptid/97fab267-c3af-11e3-b1f9-bc5ff4cad02b.eli
geli detach gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b.eli
zpool replace mixed 65487887467227878109 gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b
/usr/local/bin/sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where encrypted_provider = 'gptid/da242442-5dcf-11e3-aa6a-bc5ff4cad02b';"

Done deal. Thank you to Patrick.
 

yarox

Cadet
Joined
Feb 9, 2016
Messages
5
Does this work if there is a passphrase set which you do not know (and you also do not have a recovery key)?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Is the pool active and the volumes mounted? If so, it should work. If the pool is offline and you cannot access the encrypted data because you cannot unlock the eli devices, I fear you are out of luck.

HTH,
Patrick
 

yarox

Cadet
Joined
Feb 9, 2016
Messages
5
Thanks for answering. Already feared that.
So it seems to be the only thing to hope for is to wait until someone is able to successfully hack the encryption, whenever this may be. :-(
 

eqartimus

Explorer
Joined
Aug 2, 2014
Messages
66
This part: "Second, please do not blindly follow these instructions if you do not know what you are doing. " has me a little concerned. I have *some* notion of what your commands are doing, but my knowledge level is fairly low about this stuff.

I have backed up my Freenas server to another machine. I am about to start the above process on my FreeNAS server. Is *this* (the above post) still the best way to go about removal of encryption on my server? Is there any *gotcha* that I could be prepared for? I am a bit nervous, but of course, I am fully backed up. Though I have never tried to restore from backups ...

Thanks in advance and thanks for the original write up.
 

eqartimus

Explorer
Joined
Aug 2, 2014
Messages
66
FWIW, I'm resilvering the second drive now ... seems to be progressing without difficulties :) /finger-cross.

-update:

resilvering 3rd drive.
resilvering last drive :) /finger-cross for last steps
 
Last edited:

eqartimus

Explorer
Joined
Aug 2, 2014
Messages
66
Ok, Im at the final steps ... I am detaching the volume with the GUI. I have turned off all services ... I have shut down all jails. I still get the message:

ATTENTION - Currently the following services depend on this volume: jails

Is this important? I can't find a way to turn off that share. Do I ignore this, or is their a way to shut down jails that I can't seem to find?

Thanks in advance.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Maybe this is referring to the Jails | Configuration tab, which selects the Jail Root.
 

mahava

Cadet
Joined
Aug 18, 2016
Messages
1
Hmm this seems like a good approach, or is it? i want to remove my encryption as it is not needed anymore, i have backed up all data to another server and was about to just destroy the volume and set up again and then sync back, but this might be a better approach, i have my backup should anything go "oink", or is it possible one will run int to troubles down the road if doing this?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
The procedure is pretty straightforward. I just did not provide completely fool-proof instructions for every single step. Like "repeat for the remaining disks" - you need to know where to put the disks' IDs.

BTW: the reason why I removed encryption was the fact that you need to enter your secret manually after every reboot. One might argue that this is the purpose of encrypted disks. I would still like to have an option to store the encryption secret in the FreeNAS configuration and automount everything on boot. Why? Because in this case I can still throw a single flaky disk in the trash without erasing it first. I'm not that worried about someone stealing the entire NAS.

Kind regards,
Patrick
 
Top