SOLVED Encrypted 11.2-U3 pool can't be decrypted...

[Mr. Slumber]

Ok, here's what happened:

Still in the testing phase with one of my FreeNAS 11.2-U3 machines I have one zfs-pool: 1 vdev, raidz1, 4x 8TB HDDs, encrypted with a passphrase. For testing purposes I filled the pool with data (not more than 80%), rebooted the machine = stable for 5 days, no problems. Then for testing purposes I detached the pool (without destroying and without deleting the pool information), rebooted the machine and tried to import the decrypted pool via gui: uploaded the geli.key, entered the passphase and:

in the modern gui this showed up, nothing happens, even after some hours:

in the legacy gui I tried the exakt some but got a different error message in a pop up window:

Code:

Environment:

Software Version: FreeNAS-11.2-U3 (e140b6b8a)
Request Method: POST
Request URL: https://192.168.178.163/legacy/storage/auto-import/?X-Progress-ID=643955a9-dfbe-4d2a-8610-72b65510a5d9


Traceback:
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py" in _new_conn
  141.                 (self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py" in create_connection
  83.         raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py" in create_connection
  73.             sock.connect(sa)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in urlopen
  601.                                                   chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _make_request
  357.             conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py" in request
  1239.         self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py" in _send_request
  1285.         self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py" in endheaders
  1234.         self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py" in _send_output
  1026.         self.send(msg)
File "/usr/local/lib/python3.6/http/client.py" in send
  964.                 self.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py" in connect
  166.         conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py" in _new_conn
  150.                 self, "Failed to establish a new connection: %s" % e)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py" in send
  440.                     timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in urlopen
  639.                                         _stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py" in increment
  388.             raise MaxRetryError(_pool, url, error or ResponseError(cause))
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  163.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in dispatch
  248.         response = super(WizardView, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in post
  301.         if form.is_valid():
File "/usr/local/lib/python3.6/site-packages/django/forms/forms.py" in is_valid
  169.         return self.is_bound and not self.errors
File "/usr/local/lib/python3.6/site-packages/django/forms/forms.py" in errors
  161.             self.full_clean()
File "/usr/local/lib/python3.6/site-packages/django/forms/forms.py" in full_clean
  371.         self._clean_form()
File "/usr/local/lib/python3.6/site-packages/django/forms/forms.py" in _clean_form
  398.             cleaned_data = self.clean()
File "./freenasUI/storage/forms.py" in clean
  981.             upload_job_and_wait(key, 'disk.decrypt', disks, passphrase)
File "./freenasUI/middleware/util.py" in upload_job_and_wait
  55.         return wait_job(c, job_id)
File "./freenasUI/middleware/util.py" in upload_job_and_wait
  51.                 'Authorization': f'Token {token}',
File "/usr/local/lib/python3.6/site-packages/requests/api.py" in post
  112.     return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py" in request
  58.         return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py" in request
  508.         resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py" in send
  618.         r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py" in send
  508.             raise ConnectionError(e, request=request)

Exception Type: ConnectionError at /legacy/storage/auto-import/
Exception Value: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /_upload/ (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 61] Connection refused',))

I would really appreciate any help. Thank you in advance.

My setup:
FreeNAS 11.2_U3, Case: Intertech 4U 4129-N, Backplane: 2x ICY DOCK FatCage MB155SP-B 5 Bay, PSU: bequiet Dark Power Pro 11 550W , Motherboard: Supermicro X11SSM-F, CPU: Intel Core i3-7320,RAM: 2x 8GB ECC Ram Samsung M391A1K43BB1-CRC,HBA: LSI 9211-8i, Boot: 1x Intel SSD SSDSC2KW128G8X1 545s 128GB,

 

[PhiloEpisteme]

Then for testing purposes I detached the pool (without destroying and without deleting the pool information), rebooted the machine and tried to import the decrypted pool via gui: uploaded the geli.key, entered the passphase and:

Based on the error above I don't think this is the issue, but you were sure to download the geli.key right before you detached it or after you made any changes to the pool's encryption? You can double-check you've got the correct keys by trying to manually unlock a single drive in the pool.

 

[Mr. Slumber]

you were sure to download the geli.key right before you detached it or after you made any changes to the pool's encryption?

Yes! After I setup the encrypted pool I downloaded the geli.key immediately and never changed anything afterwards. So the key should definitely work. Same for the passphrase.

You can double-check you've got the correct keys by trying to manually unlock a single drive in the pool.

Did that. Also doesn't work. With the legacy gui I also get the same error message.

Interesting thing is that this doesn't happen with my 11.1U7 test machine...

So, what should I do? It's not a good feeling to know if there are maybe problems in the future with my pool and I have to reboot the machine I will not be able to decrypt the pool again.

 

[Mr. Slumber]

Just to clear things up: I have the geli.key (mentioned above) and the passphrase. What I do not have is a recovery key. A recovery key is optional if you have your "normal" key and passphrase, right?

Thanks for your ideas.

 

[Mr. Slumber]

Based on the error above I don't think this is the issue

Ok but doesn't this part of the error message point to a "connection problem"? Sorry I'm not a coder.

Code:

Exception Value: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded
with url: /_upload/ (Caused by NewConnectionError(':
Failed to establish a new connection: [Errno 61] Connection refused',))

 

[Mr. Slumber]

Also the message "undefined" (see screenshot below) immediately appears in the console after I hit "next" (see screenshot below) in the modern gui...

 

[PhiloEpisteme]

So, what should I do? It's not a good feeling to know if there are maybe problems in the future with my pool and I have to reboot the machine I will not be able to decrypt the pool again.

If you have the correct key you will be able to unlock your drives provided there isn't any extremely unfortunate drive corruption.

I have the geli.key (mentioned above) and the passphrase. What I do not have is a recovery key. A recovery key is optional if you have your "normal" key and passphrase, right?

Check out this post about geli encryption and how it is used by FreeNAS. The absolute safest thing is to ALWAYS keep both the geli.key and recovery.key and back them up in multiple places every time you make any alterations to your encryption or drives. You need at least one of them with the password if set; do note the recovery key does not have password support through FreeNAS.

Yes! After I setup the encrypted pool I downloaded the geli.key immediately and never changed anything afterwards. So the key should definitely work. Same for the passphrase.

Did you try to manually unlock the drives to confirm the key is correct or are you just relying on having downloaded the correct key? I doubt this is the issue given the messages but lets make absolutely certain it is the correct key by testing it against each drive if you haven't already. Depending on what you may have done to the pool the key may not necessarily be the correct key anymore.

192.168.178.163

Is this the IP of the machine you are using to test or another machine on your network?

To encourage more responses would you mind following the Forum Rules and either create a new post or edit your first post containing the exact software and firmware versions and exact hardware you're using. It'll encourage folks with more knowledge to join the discussion. :)

 

[Mr. Slumber]

Ok, I solved my problem:

I thought it was an excellent idea to safe my geli.key to my 1Password Vault. NO, it is not!!!
Don't try this @home, it'll f... up destroy your keys. Why? Because as I found out in the 1Password forum the actual version of 1Password 7.2.5 can import files/documents but it is not able to export these kind of "documents" (or better call it files). You can test it with other types of files (pdf, ppt e.g): import yes, export is possible (MAC/Win) but the exported file (as long as it is not a plain txt file) is badly damaged... So thanks again 1Password, again a big troublemaker in my software arsenal.

TL;DR: never save your keys or config files via 1Password it'll generate them useless!

Glad I learned this lesson before this machine went into production.

 

[PhiloEpisteme]

Because as I found out in the 1Password forum the actual version of 1Password 7.2.5 can import files/documents but it is not able to export these kind of "documents" (or better call it files). You can test it with other types of files (pdf, ppt e.g): import yes, export is possible (MAC/Win) but the exported file (as long as it is not a plain txt file) is badly damaged... So thanks again 1Password, again a big troublemaker in my software arsenal.

Do you have a link to the post in question? That seems like an incredible flaw in 1Password's product.

 

[Mr. Slumber]

Do you have a link to the post in question?

Quiet some in the 1Password forums, just look for "export file".
The "best" thing is that they updated their support export document. It clearly states that you can only export "*.pif" and "*.csv" (MAC) and "*.csv" and "*.txt" "data" (Win) (why not call it files?!).

Ok, my fault, I didn't read this document before saving my keys but... :(

 

[Mr. Slumber]

Also 1Password shows a pop up window if you try to export e.g. your geli.key:

Sorry, german, so quick translation:

Export of documents is not supported

Some of the objects you tried to export are documents. 1Password doesn't support the export of documents yet.

 

[Apollo]

TLDR,

The reason it crashes Python is because the information related to the pools is already part of the system.
Shut down your system, attach the pool, power-up and the pool will be presented to you, but still in the locked state until you unlock it with the passphrase.
There was a bug or deficiency when you had other pools removed but not detached and importing a new pool from scratch would cause the same type of errors. This is no longer the case with the latest FreeNAS-11.2-U3 .

 

[Mr. Slumber]

Jesus H. Christ, it's still not working...

Here's what I did: fresh install of 11.2-U3. Setup of an encrypted pool (1 vdev, raidz1, 4 HDDs), populated it with 1TB of data, created a passphrase. Now I downloaded the key, the recovery key and the config file of the machine and saved it locally (of course not via 1Password ;) ).

I locked the pool, detached it, rebooted the machine and tried to import the pool again (with the key and passphrase = errors see above; with the recovery key and passphrase = errors see above)... Still the exact the errors as mentioned above... Noooooo

I really appreciate your help cause this renders FreeNAS 11.2-U3 useless for me for real worse case scenarios... Thank you in advance :)

 

[SweetAndLow]

Jesus H. Christ, it's still not working...

Here's what I did: fresh install of 11.2-U3. Setup of an encrypted pool (1 vdev, raidz1, 4 HDDs), populated it with 1TB of data, created a passphrase. Now I downloaded the key, the recovery key and the config file of the machine and saved it locally (of course not via 1Password ;) ).

I locked the pool, detached it, rebooted the machine and tried to import the pool again (with the key and passphrase = errors see above; with the recovery key and passphrase = errors see above)... Still the exact the errors as mentioned above... Noooooo

I really appreciate your help cause this renders FreeNAS 11.2-U3 useless for me for real worse case scenarios... Thank you in advance :)

You file a bug yet? It's pretty obvious that it just doesn't work and you have reproduce it twice now.

 

[Mr. Slumber]

You file a bug yet?

Ok, I'll do that (have to eduacte myself before on how to do this the correct way ;)) . I think I'll switch back to 11.1-U7 now.

 

[Mr. Slumber]

For those who are interested:

Did a fresh install of 11.1-U7 on this machine!

Setup of an encrypted pool (1 vdev, raidz1, 4 HDDs), populated it with 1TB of data, created a passphrase. Now I downloaded the key, the recovery key and the config file of the machine and saved it locally (of course not via 1Password ;) ).

I locked the pool, detached it, rebooted the machine and tried to import the pool again... yessss, it just works!
Will definitely stay with 11.1-U7.

 

[Apollo]

Jesus H. Christ, it's still not working...

Here's what I did: fresh install of 11.2-U3. Setup of an encrypted pool (1 vdev, raidz1, 4 HDDs), populated it with 1TB of data, created a passphrase. Now I downloaded the key, the recovery key and the config file of the machine and saved it locally (of course not via 1Password ;) ).

I locked the pool, detached it, rebooted the machine and tried to import the pool again (with the key and passphrase = errors see above; with the recovery key and passphrase = errors see above)... Still the exact the errors as mentioned above... Noooooo

I really appreciate your help cause this renders FreeNAS 11.2-U3 useless for me for real worse case scenarios... Thank you in advance :)

Let me guide you through the process, just to make sure you didn't miss a thing:


1) You create the encrypted pool on a fresh new install with passphrase, saved the key and recovery key.(You did it and it works).

2) You don't need to locked the pool and then detach it. You just detach it if you want, otherwise, if you intend to power your system off, best to just proceed to power down without locking nor detaching as in "Pool detach" command. When system is off, you can them remove your disks from the system. I wish ZFS and Freenas would have provided a more meaningful name for to differentiate between both types of "Detach".

So to summarize, you told Freenas to detach the pool which will un-mount the pool and remove the keys on the Freenas boot drive.
When you selected "Detach", what option did you use? Did you ask to clear the content of the disks by mistake?
Actually with the new GUI, the term has been reworded as "Export/Disconnect" which is better.

3) After rebooting/power system ON, with the drives physically connected to the system:
Under Storage => Pools:
- Select and press: "ADD"
- Select: "Import an existing pool"
- Press "NEXT".
- Select option for encrypted pool: "Yes, decrypt the disks". Extra options related to encrypted pool will appear.

- Select: "Encryption Key" => "Browse..." and point to your "Geli.key" file you saved earlier.
- Press "Open" in explorer.
- Select: "UPLOAD"

- Either the Disk selection gets populated or not, the disk should be selected ( in this case 4 of the disk you have should be present)
- Enter the "Passphrase"
-Select: "NEXT"

From memory, you will be given the pool name, you need to select it and proceed with the import.
If this appens, it means the key has been correctly recognized and the data can be decrypted.

At this point, you should get "Sarting job..." or something.
There shouldn't be any Python related errors at this point. Just let is proceed.
You can open SSH and do "zpool status" and if decryption works, the pool would be made available.

If this doesn't work, skip regular GELI.KEY and select the recovery key. At this point it will not ask for the passphrase.

When the import is in progress, it will take a while for the datasets to be mounted (it can easily take 30 minutes or more) . If you have a monitor or IPMI connected, you can follow the progress on the console.

 

[SweetAndLow]

- Either the Disk selection gets populated or not, the disk should be selected ( in this case 4 of the disk you have should be present)

EDIT: I did more testing.

I tried to lock pool and then unlock using passphrase: This works
I tried to lock pool and reboot then unlock using passphrase: this works
I tried to export/Dissconnect pool then import, this requires uploading the geli.key: this works

I'm confused to what the OP is doing now, because my testing shows that it's working just fine. I have very little experience with encrypted pools.

 

[Apollo]

EDIT: I did more testing.

I tried to lock pool and then unlock using passphrase: This works
I tried to lock pool and reboot then unlock using passphrase: this works
I tried to export/Dissconnect pool then import, this requires uploading the geli.key: this works

I'm confused to what the OP is doing now, because my testing shows that it's working just fine. I have very little experience with encrypted pools.

I just noticed, as I am going through the process myself to provide answers, that in the OP first post, the "Disks*" line above the "Encryption Key" option doesn't contains the list of the disk that need to be imported and decrypted.

 

[Mr. Slumber]

An update for those who are interested:

I think I solved this in my case but I don't know if this will work out for someone else but I wanted to share this ;)

So as I found out it worked for me with 11.1-U7.
Now I did some fresh installs of 11.2-U3 again and again. In case A) I only used the modern gui. In case B) I only used the legacy gui. And in case C) I used the modern and the legacy.
So I learned that only in case A) (using strictly only the modern gui) I could reattach/import and decrypt my detached pool. In this case it really works :) If using case B) or C) I always got the errors posted above and never got it to work.


TL;DR: I went back to 11.2-U3, never used the legacy gui and everything works.

(Seems strange but this was the only way to get this running!)

Thanks to everyone for your help and support! :)