Exception Type: MiddlewareError at /storage/volume/2/unlock/

[devnullius]

I had a corrupt USB boot device. I installed FreeNAS on a fresh stick and imported my settings (including seeds for the passwords).

Today, I wanted to check some info I have stored and went to unlock my volume. I got this instead (rather quickly too, after entering the password):

Code:

Environment:

Software Version: FreeNAS-11.1-U7 (b45bfcf29)
Request Method: POST
Request URL: http://192.168.0.197/storage/volume/2/unlock/?X-Progress-ID=e9e57334-1890-496a-8c75-ad41fe54f073


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volume_unlock
  1036.             form.done(volume=volume)
File "./freenasUI/storage/forms.py" in done
  2873.             raise MiddlewareError(msg)

Exception Type: MiddlewareError at /storage/volume/2/unlock/
Exception Value: [MiddlewareError: Volume could not be imported: 8 devices failed to decrypt]

All 8 disks are up and running: no problems there as far as I can tell. My emailed reports don't mention pending updates since (lastly) December 27th, so I assume I was on the 11.1U7 train before and after the failing boot device. Correction: I was on U6 when I made the backup and I restored the settings backup to U7. No errors were given.

 

[PhiloEpisteme]

Exception Value: [MiddlewareError: Volume could not be imported: 8 devices failed to decrypt]

It looks like you were using encrypted drives.

The decryption keys are not stored in the config backups, they have to be saved separately. Did you save the decryption keys? If not, how corrupt is your old USB boot device? Even if you cannot boot from it can you still read data from it? If you have the decryption keys backed up as well or if you can read the data on that old USB drive you're probably alright and I (or someone else) can likely help you get access to your drives again. If you don't/can't you may have lost access to your data.

 

[devnullius]

wow - for real?? I should still have the key lying around but wow - that's DANGEROUS when the general advise is not to rebuild a boot pool but instead start fresh. Especially when the export mentions including the passwords for you… So without the key I'd have lost +10TB of data just like that?

Gonna test next :) But wow... ;-(

Thanks!


PS: the old USB boot disks have all been fully wiped (no errors found) and I switched from an USB boot pool to a new USB boot disk. They encryption key for the pool was saved ;-)

 

[devnullius]

Update: using the decryption key gives the exact same error (I even tried AND password AND decryption key).

So it looks like a bug of some sorts…? (https://www.ixsystems.com/community...-pool-throws-exception-middlewareerror.60230/) (https://redmine.ixsystems.com/issues/27529 - I do have this Critical error so I'll see to get that fixed first too: " These ports are not ACTIVE on LAGG interface lagg0: em1. Please check cabling and switch. (Fixed)".

 

[PhiloEpisteme]

wow - for real?? I should still have the key lying around but wow - that's DANGEROUS when the general advise is not to rebuild a boot pool but instead start fresh. Especially when the export mentions including the passwords for you… So without the key I'd have lost +10TB of data just like that?

The documentation is not very good in this regard, I think. It is for security reasons that your config backups do not contain your decryption keys. If it did, anyone with access to your config backup would be able to access any data on your encrypted disks.

Have you double-checked that your encryption key is the correct one? You can test it using basic geli commands from the command line. I am only suggesting you deviate from using the UI to better confirm whether your keys are the correct ones. geli attach -k <geli_key_file> <disk_to_unlock>

 

[devnullius]

(will test next - thanks)

The console log doesn't look inspiring though ;-(

Scroll towards the end please… I included boot logs too: https://pastebin.com/5D5LMFkd

 

[devnullius]

Have you double-checked that your encryption key is the correct one? You can test it using basic geli commands from the command line. I am only suggesting you deviate from using the UI to better confirm whether your keys are the correct ones. geli attach -k <geli_key_file> <disk_to_unlock>

So I should copy my decryption key somehow to my FreeNAS? I'll take a look if I can 'cat' it and I'll edit below. That said, the key should be good… I hope ;-(

 

[PhiloEpisteme]

So I should copy my decryption key somehow to my FreeNAS? I'll take a look if I can 'cat' it and I'll edit below. That said, the key should be good.

Yeah, however you can get the key onto your system is fine. When I started using FreeNAS I went through tons of tests on how to use the keys, back them up, restore them etc.

Looking at the log file you posted it looks like your new system is trying to do the right thing.

Mar 25 15:18:36 Freenas uwsgi: [middleware.exceptions:36] [MiddlewareError: Unable to geli attach gptid/7a2fc1c9-08ef-11e8-ab60-0025901159d4: geli: Cannot open keyfile /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key: No such file or directory.

It is even giving you a nice hint of how you should restore your encryption keys.

Do you know if you have both keys backed up? The FreeNAS terminology is little different than straight Geli terminology. In FreeNAS terms, do you have the encryption key AND the recover key backed up? In Geli terminology do you have both User Key 1 and User Key 2 backed up?

If you only have one backed up that is okay. There just may be an extra step to get you full back where you want to be.

 

[devnullius]

This is my filename…
FreeNAS 8x 3TB NAS private key encryption decypher with default wallet password behind it ;-) geli.key
If I open that file in Notepad++ it looks more like Binary than anything else. Does this help?

Looks like I was on FreeNAS 11.1-U6 though, NOT *-U7. Could that be it too?
"Freenas-FreeNAS-11.1-U6 (caffd76fa)-20190313112133.tar" is the backup file for my settings before I replaced the boot disk.

 

[PhiloEpisteme]

FreeNAS 8x 3TB NAS private key encryption decypher with default wallet password behind it ;-) geli.key
If I open that file in Notepad++ it looks more like Binary than anything else. Does this help?

Cool, download that geli.key to your computer and then copy it to your sever. You can use scp for that, something like scp geli.key <user>@<freenas ip>:~/.geli.key where the user may be root and the freenas-ip is specific to your system.

If you don't have SSH access set up on your server you'll want to set that up. Basically you'll need to enable the ssh service on your freenas box and then add your public ssh key to the appropriate user, probably the root user.

Looks like I was on FreeNAS 11.1-U6 though, NOT *-U7. Could that be it too?
"Freenas-FreeNAS-11.1-U6 (caffd76fa)-20190313112133.tar" is the backup file for my settings before I replaced the boot disk

I don't think that is an issue.

 

[devnullius]

The results are in :(

Code:

root@Freenas:~ # geli attach -k geli.key ZFS_8x_3TB_RAIDz2_pool
geli: Cannot open ZFS_8x_3TB_RAIDz2_pool: No such file or directory.

For Googlers on Windows... WinSCP does the job.

 

[PhiloEpisteme]

The results are in :(

Have no fear, I think what you actually want is something more like geli attach -k geli.key /dev/<device>. Can you provide some more information about your system? Your signature suggests you have 8 3TB drives. What is your current boot pool? How are they all connected? All via you're 9211 HBA or a mix of that and your onboard SATA?

Edit: Try using geom disk list. What we really care about is the Name and which of those disks are your encrypted disks, which are your boot pool, and which are something else.

 

[devnullius]

At least you are still hopeful :)

USB boot disk. I had a boot pool of USB disks and thought just to skip it from now on as being a waste of time (or so I was led to believe).

Then a single 10TB disk as a volume. And the Z2 pool with the 8 disks. And yes, the pool is properly connected through 9211 :)

 

[devnullius]

Code:

root@Freenas:~ # geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ATA WDC WD30EFRX-68E
   lunid: 50014ee003c0a9c1
   ident: WD-WMC4N2081759
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: ATA WDC WD30EFRX-68E
   lunid: 50014ee20c340145
   ident: WD-WCC4N4SCDS21
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: ATA WDC WD30EFRX-68E
   lunid: 50014ee05915f269
   ident: WD-WMC4N2146579
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255

Geom name: da3
Providers:
1. Name: da3
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r1w1e2
   descr: ATA MB3000EBUCH
   lunid: 5000cca225f2b1fc
   ident: YHKLJ61A
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da4
Providers:
1. Name: da4
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r1w1e2
   descr: ATA MB3000EBUCH
   lunid: 5000cca225f176df
   ident: YHKHU7UA
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da5
Providers:
1. Name: da5
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r1w1e2
   descr: ATA MB3000EBUCH
   lunid: 5000cca225d6e6f6
   ident: YHHMBURA
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da6
Providers:
1. Name: da6
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r1w1e2
   descr: ATA MB3000EBUCH
   lunid: 5000cca225f24da0
   ident: YHKKNG8A
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da7
Providers:
1. Name: da7
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r1w1e2
   descr: ATA MB3000EBUCH
   lunid: 5000cca225f24e55
   ident: YHKKNN3A
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da8
Providers:
1. Name: da8
   Mediasize: 15518924800 (14G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: TOSHIBA TransMemory
   ident: 54B80A3F9376C1218002C39C
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da9
Providers:
1. Name: da9
   Mediasize: 7743995904 (7.2G)
   Sectorsize: 512
   Mode: r0w0e0
   descr: USB DISK 2.0
   ident: 070D37A832479267
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 10000831348736 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: WDC WD101KRYZ-01JPDB1
   lunid: 5000cca266ed51d8
   ident: 7JK6PLTC
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: PLEXTOR DVDR   PX-880SA
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

root@Freenas:~ #

 

[devnullius]

Copy paste from the GUI:

Code:

Name    Used    Available    Compression    Compression Ratio    Status    Readonly    Comments
VOLU10TB 3.3 TiB (36%) 5.7 TiB -                 -                                 HEALTHY
ZFS_8x_3TB_RAIDz2_pool Locked Locked    -                                    -        LOCKED

 

[PhiloEpisteme]

At least you are still hopeful :)

Quite hopeful so long as the key you have is the correct key for your drives.

In general I think you might find this post about Geli encryption helpful.

Basically, FreeNAS uses Geli to encrypt entire disks. Geli stored the master decryption key on every disks (it is unique to each disk). That master key is itself encrypted with up to 2 User Keys, User Key 1 (the main key in FreeNAS) and User Key 2 (the Recovery Key in FreeNAS). Either can be used to decrypt the master key and thus the drive. FreeNAS sets it up such that the same pair of User Keys decrypts the master keys for all drives across a pool. This saves you from having to type 8 passwords for an 8-drive pool.

So, we should try the following steps. These assume that geli.key is in the same directory you are entering your commands from.

1. Hope that everything is perfectly okay and that you just need to restore your main encryption key to the proper location so that FreeNAS can find it. That location was in the log file above and it appears to be /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key. Copy your geli.key file to that location with that exact name. Make sure that the permissions are correct on the file. They should be -rw-r--r--.
2. If the above fails, you'll want to double-check your key. Based on the info you provided above my guess is that you can test this with geli attach -k geli.key /dev/da0. If this works, we can continue with other steps to at least give you access to your data.

 

[devnullius]

New problem: see https://www.ixsystems.com/community...nstallation-fails-because-its-too-full.75047/

I so hate FreeNAS but I do love your help up to this point, Philo! At least we get the encryption keys part correct again. But writing the new keys to disks seems to fail because there's only 4% of free space? For now, THANK YOU despite the new problem :)

 

[PhiloEpisteme]

Can you clarify what steps you took? It sounds like the decryption steps above worked but that you now have an issue with the volume being too full?

 

[devnullius]

Can you clarify what steps you took? It sounds like the decryption steps above worked but that you now have an issue with the volume being too full?

Exactly: I just made a new topic for it and finished editing all the links to https://www.ixsystems.com/community...nstallation-fails-because-its-too-full.75047/

 

[devnullius]

Quite hopeful so long as the key you have is the correct key for your drives.

In general I think you might find this post about Geli encryption helpful.

Basically, FreeNAS uses Geli to encrypt entire disks. Geli stored the master decryption key on every disks (it is unique to each disk). That master key is itself encrypted with up to 2 User Keys, User Key 1 (the main key in FreeNAS) and User Key 2 (the Recovery Key in FreeNAS). Either can be used to decrypt the master key and thus the drive. FreeNAS sets it up such that the same pair of User Keys decrypts the master keys for all drives across a pool. This saves you from having to type 8 passwords for an 8-drive pool.

So, we should try the following steps. These assume that geli.key is in the same directory you are entering your commands from.

Update on this problem: I created a freshly formatted new FreeNAS 11.1-U6 (NOT 7) boot disk (PS: I was even willing to go test with v9.10, see if that could've handled it, per some 2017 threads). After booting with the bootdisk I entered my way through the initial setup wizard. It only found my 10TB pool (single disk), not my real pool.

So I went to Volume Manager - Import Volume, had it do it's things and entered my geli key and my password. It found 8 disks and after selecting them all the pool opened without any further problems, even at 96% full! :)

Now my question... Would it be really dumb to import my old settings again? Because I made a lot of user accounts & Windows shares and I hate to see them gone. It was a real pita to set it up, once :)

THANK you to all the nice peeps willing to help me out here each and every step of the way. I even got a personal message yesterday from PhiloEpisteme whom especially kept the thread alive and moving forward!

It was a stressful and frustrating week for sure. I still don't know what triggered the problems in the first place? Maybe importing 11.1-U6 settings into a 11.1-U7 environment? Shall I proceed with Importing Them Settings once more, now I'm back to the 11.1-U6 train?

xxx

Hey,

I did some experimenting on my machine and verified that I was able to decrypt my disks manually and import them with the following commands. It is similar to what we worked out in our various messages. In the example below you will have to substitute the path to your geli key, the gptid devices, and the pool name with the correct values for your situation. I hope we can import your pool and get your data safe!


$ geli attach -k encryption.key /dev/gptid/4bd4617b-52e4-11e9-b3d1-002590d4322f $ geli attach -k encryption.key /dev/gptid/501af881-52e4-11e9-b3d1-002590d4322f $ zpool import $ zpool import -R /mnt pool



Other folks on the forums have suggested that you can replace your drives with larger drives to gain space to decrypt. I haven't yet had the chance to experiment with that or to research how to accomplish that if you cannot import your pool. If the above does not work I think exploring this option makes sense.

PS: still not sure if that last part would've held true... We could decrypt the disks, yes, but not "mount" it or import it properly. And if you can't do that... How can you give commands to that pool? Seems wrong to me :) But what do I know when it comes to FreeNAS ;p