FreeNAS 11.1-U7 Cannot import encrypted volume

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
FreeNAS 11.1-U7 Cannot import encrypted volume

The story goes:
I'm running a 4 disk (3TB each) RAIDZ2 configuration & I was replacing an overheating disk. Shut the system down, removed the disk, put a new one in, and restarted. It's a little vague right now, as it was a couple of days ago, but it recognized, yet couldn't add the replacement disk for some reason.
Long and short is that the volume showed up as locked in the GUI, and I couldn't get it to unlock (I have the geli.key). I read on a forum somewhere that unmounting the volume and then reattaching it sometimes solved the problem. I tried this, and now I can't get the volume to attach/decrypt. The following is what I'm doing, trying to get it to mount. I have an external drive I can copy everything to in one shot and redo the entire NAS but, obviously, I need to get to the data first...

Log into the GUI:
Volumes --> Import Volume --> "Encrypted ZFS volume?" I select "Yes: Decrypt disks" and click ok.
Screen Shot 2019-05-21 at 18.04.51.png
I can choose ada0p2, ada1p2, ada2p2, and/or ada3p2. Through trial and error, I have found that I can choose ONLY 2 disks (I can select all 4 to "use" at once, but it only works when I select as noted), and that for some reason, one of them MUST be ada3p2 (If I choose any other combination, "Step 3" comes up, however, there is no volume to select in the drop down menu.
Screen Shot 2019-05-21 at 17.29.34.png
For the sake of consistency in this post, I'll stick to selecting ada2p2 and ada2p3 each time). So, I then choose the encryption key, and as I don't believe I set a passphrase, I hit ok.
Screen Shot 2019-05-21 at 18.05.28.png
It thinks for a moment and comes up with step 3, with a selectable volume in the drop down menu.
Screen Shot 2019-05-21 at 18.05.56.png
I select it, and click ok, and it thinks for a few seconds, and I get the following error message:
Screen Shot 2019-05-21 at 18.13.25.png
Code:
Request Method:    POST
Request URL:    http://192.168.1.199/storage/auto-import/
Software Version:    FreeNAS-11.1-U7 (b45bfcf29)
Exception Type:    MiddlewareError
Exception Value:   
[MiddlewareError: The volume "NAS-Root" failed to import, for futher details check pool status]
Exception Location:    ./freenasUI/middleware/notifier.py in volume_import, line 2676
Server time:    Tue, 21 May 2019 17:50:14 -0400
Traceback
Environment:

Software Version: FreeNAS-11.1-U7 (b45bfcf29)
Request Method: POST
Request URL: http://192.168.1.199/storage/auto-import/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in dispatch
  237.         response = super(WizardView, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in post
  300.                 return self.render_done(form, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in render_done
  357.                                   **kwargs)
File "./freenasUI/storage/forms.py" in done
  884.         self.volume = notifier().volume_import(vol['label'], vol['id'], key, passphrase, enc_disks)
File "./freenasUI/middleware/notifier.py" in volume_import
  2676.                     'for futher details check pool status') % volume_name)

Exception Type: MiddlewareError at /storage/auto-import/
Exception Value: [MiddlewareError: The volume "NAS-Root" failed to import, for futher details check pool status]



Request information
GET
No GET data

POST
Variable    Value
auto_import_wizard-current_step    '2'
2-__all__    ''
2-volume_id    'NAS-Root|14887166266537682792'
__form_id    'dijit_form_Form_99'
FILES
No FILES data

COOKIES
Variable    Value
fntreeSaveStateCookie    'root'
csrftoken    '********'
sessionid    'pd7va30b3ldh113zadubqz7xbgtd2y1t'
META
Variable    Value


I have both "Show console messages in the footer" selected in advanced preferences, as well as a monitor attached, and the following is what comes up when I try the above:
Code:
May 21 17:49:36 freenas GEOM_ELI: Device gptid/bec972c2-9131-11e8-991f-3860770e8bb4.eli destroyed.
May 21 17:49:36 freenas GEOM_ELI: Device gptid/008eab85-d6e2-11e8-80d6-3860770e8bb4.eli destroyed.
May 21 17:49:45 freenas GEOM_ELI: Device gptid/bec972c2-9131-11e8-991f-3860770e8bb4.eli created.
May 21 17:49:45 freenas GEOM_ELI: Encryption: AES-XTS 256
May 21 17:49:45 freenas GEOM_ELI:     Crypto: software
May 21 17:49:45 freenas GEOM_ELI: Device gptid/008eab85-d6e2-11e8-80d6-3860770e8bb4.eli created.
May 21 17:49:45 freenas GEOM_ELI: Encryption: AES-XTS 256
May 21 17:49:45 freenas GEOM_ELI:     Crypto: software
May 21 17:49:55 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=5236072806288514073
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=8735114228111867127
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=14678466695872316278
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=17447851299479880634
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=9336531035625982523
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=5236072806288514073
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=8735114228111867127
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=14678466695872316278
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=17447851299479880634
May 21 17:49:56 freenas ZFS: vdev state changed, pool_guid=14887166266537682792 vdev_guid=9336531035625982523
May 21 17:49:56 freenas uwsgi: [middleware.notifier:2553] Importing NAS-Root [14887166266537682792] failed with: cannot import 'NAS-Root': I/O error
May 21 17:49:56 freenas uwsgi: [middleware.exceptions:36] [MiddlewareError: The volume "NAS-Root" failed to import, for futher details check pool status]
May 21 17:50:13 freenas nfsd: can't register svc name


PLEASE someone help me - I have a LOT of crap on there, but I do have some important family photos, & things.

Here's my setup:
Motherboard: HP p6-2000z (1.6GHz AMD)
RAM: 4GB (YES, I know the minimum is supposedly 8GB, but it's been running perfectly for ages now)
Storage: 4x 3TB Seagate Constellation drives
System drive: 8GB flash drive
Screen Shot 2019-05-21 at 18.00.40.png Screen Shot 2019-05-21 at 18.07.06.png


(A follow up - provided we're successful here {and even if we aren't} - can anyone link me to somewhere that there is a list of commands to enter in the shell to copy to an externally plugged in USB drive & back, please?)


Thank you ALL for your time, just to read it, even if you can't help me!
 

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
you might be able to try importing from the other GUI, I believe it was at least partially available in 11.1.
alternatively, it should be possible to import from the commandline, since it seems like you are having trouble with the GUI.
iirc, you use something like "geli -k /key /dev" to decrypt the drives and then zfs import to import your pool, at which point it will fucntion as a normal zpool, albeit the GUI might not pick it up completely.

you can use replication to clone your pool, once imported, to a USB drive that has a zfs pool. mounting a drive with anything but zfs would be very manual.
alternatively, you can use rsync or cp to copy from any mounted filesystem to any other, with rsync being far more robust and recommended generally.

you also might submit this as a bug report, one of the devs may be able to help you fix it there.

you might also consider, as long as you have your geli.key (make sure you back this up somewhere!), creating yourself a new install on a different boot drive, and trying to import that way, possibly bypassing whatever the problem is.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
When a pool is locked, to be able to unlock it requires entering the passphrase.
As you stated, if your pool was made of 4 disks, then you will need to decrypt at least 3 out or the 4 but your pool will be degraded.
I suspect the reason it fails the import is that you have only provided 2 out of the 3 or 4 disks.
I would put back the overheating disk to the pool and try to import again.
If you have a recent backup of your freenas config, you may want to try using it on a different freenas install.
If you recently upgradeed, you would want to reboot using the previous boot environment. Most safest way to go I think.

When you do replace a disk from an encrypted pool, you need to have a passphrase for the change to happen and then regenerate a key.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
you might be able to try importing from the other GUI, I believe it was at least partially available in 11.1.
alternatively, it should be possible to import from the commandline, since it seems like you are having trouble with the GUI.
iirc, you use something like "geli -k /key /dev" to decrypt the drives and then zfs import to import your pool, at which point it will fucntion as a normal zpool, albeit the GUI might not pick it up completely.

you can use replication to clone your pool, once imported, to a USB drive that has a zfs pool. mounting a drive with anything but zfs would be very manual.
alternatively, you can use rsync or cp to copy from any mounted filesystem to any other, with rsync being far more robust and recommended generally.

you also might submit this as a bug report, one of the devs may be able to help you fix it there.

you might also consider, as long as you have your geli.key (make sure you back this up somewhere!), creating yourself a new install on a different boot drive, and trying to import that way, possibly bypassing whatever the problem is.

So, I looked into using the other GUI, and apparently it's still technically accessible, but you need to enter the direct URL to access it, because it is apparently very buggy and not really meant to be used.
Frankly, using the command line TERRIFIES me. I looked around a bit and couldn't find a direct list of commands (I'm a bit of a newbie to this, but not a complete idiot). If I had a backup of the important data, I'd try it no problem, but I really want to only try it if I have to. But I had a thought - b/c I have it set up as RAIDZ2, I can lose up to 2 disks. As I have 4, can I simply physically disconnect two of them for my own peace of mind while doing this?

As far as copying the data off, that's much less of an issue than just accessing it. I can just copy it over my network. I just thought it might be easier if I just stuck in a drive. So primarily, it's still just getting the data accessible - even just for a single boot, then I'll copy it off & just do a fresh install.

To submit it as a bug report, would I have to repost it all in the bug section, or can I just put a new tag on this one?

Also, I did try doing a clean install on a separate drive, and same issue
you might be able to try importing from the other GUI, I believe it was at least partially available in 11.1.
alternatively, it should be possible to import from the commandline, since it seems like you are having trouble with the GUI.
iirc, you use something like "geli -k /key /dev" to decrypt the drives and then zfs import to import your pool, at which point it will fucntion as a normal zpool, albeit the GUI might not pick it up completely.

you can use replication to clone your pool, once imported, to a USB drive that has a zfs pool. mounting a drive with anything but zfs would be very manual.
alternatively, you can use rsync or cp to copy from any mounted filesystem to any other, with rsync being far more robust and recommended generally.

you also might submit this as a bug report, one of the devs may be able to help you fix it there.

you might also consider, as long as you have your geli.key (make sure you back this up somewhere!), creating yourself a new install on a different boot drive, and trying to import that way, possibly bypassing whatever the problem is.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
When a pool is locked, to be able to unlock it requires entering the passphrase.
As you stated, if your pool was made of 4 disks, then you will need to decrypt at least 3 out or the 4 but your pool will be degraded.
I suspect the reason it fails the import is that you have only provided 2 out of the 3 or 4 disks.
I would put back the overheating disk to the pool and try to import again.
If you have a recent backup of your freenas config, you may want to try using it on a different freenas install.
If you recently upgradeed, you would want to reboot using the previous boot environment. Most safest way to go I think.

When you do replace a disk from an encrypted pool, you need to have a passphrase for the change to happen and then regenerate a key.

Yes, the pool was/is made of 4 disks, but it was a RAID Z2, which means I SHOULD be able to lose up to 2 disks, and (although in a degraded state) the pool would still function & the data would be there & accessible. Now, doesn't that mean I should be able to recover the pool with just 2? Or is there some reason I would need 3?

I have put back the overheating disk & the issue persists.

Unfortunately, due to a series of WILDLY unbelievably coincidental and HORRIBLY timed events, I only have some of the data backed up - but not the most important stuff (still double checking, but haven't found it in some dark corner of my hard drives, yet).

I did not recently upgrade. I've been (successfully & without incident) running this version (U7) since shortly after it was released, and I've never run anything below 11.x.

Well, I DO have the key, but I don't remember ever entering a passphrase. I've tried the same as my log in pw to the web gui, and a few others I can think of that I'd use, but none work.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Not to scare you in any way, but you are running on thin ice.
With encrypted disks, the challenges are a bit more problematic.
The reason I believe you have problem with your pool might have been due to the lack of passphrase as you seem to suggest. If you hadn't restarted your Freenas box in a while then the issue might have been there since day one and never saw it coming. I suspect you did an upgrade of Freenas and between 10 and 11 or so, the encryption algorithm has changed and also I believe due to the presence of a bug somewhere along the line.
With Coral many people have lost there encrypted pool due to such issue, myself included, but I had my pool replicated to several backups.
One of the issue with encryption was the inability of installing the redundant disk of a RAIDZ2 array. On a 5 disk RAIDZ2, the 2 dedundant disks would be made UN-available.

I believe in order to fix the issue I had to have passphrase set for the encrypted pool. Without a passphrase, the pool would be allowed to restore redundance, but only until the next reboot. The lack of passphrase would cause the algorithm to prevent restoring or updating the key and the pool would lost make the disk unavailble once more.

Without redundancy as you are, there is a chance you can loose your entire pool. Hence the reason why you are running on thin ice.
Also, this is not a position I like being put into, ( I mean me trying to giving you advice knowing what ever you will do can end up as a disaster.)


There re several scenario you should be made aware of in your actual situation. Rememebr you don't know what part of your data is backed up:

The more time you leave your system running in a degraded state with encrypted pool, the more chances of facing problems.

To that end here are the scenarion:

1) you leave the pool as is and you try to backup what you have. You can make things worse depending on your backup strategy, and it might take a while.

2) You can try replacing the disks advertised as unavailable. To do that you will have to follow the encrypted pool disk replacement. Be careful as not to add the disk to the pool. If you do you will be in deep troubles.
You should assign the 2 physical disk that are no longer in use (the one that were not able to be attached again.
If it lets you proceed witht the replacement and resilvering is underway, you will need to create a passphrase to the pool as resilvering is taking place. If you don't, the disk will be made unavailable at the end and you would have wasted valuable time.
You will need to save the GELI key and the Recovery GELI key to be safe.
I believe, once the passphrase has been created, it can be removed on the old GUI without too much hope of failing the recovery. Mind you I would at least keep it until the resilvering is complete and successful.

3) I have no more relevant solutions. Running a scrub on the degraded pool would be a waste of time.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
Not to scare you in any way, but you are running on thin ice.
With encrypted disks, the challenges are a bit more problematic.
The reason I believe you have problem with your pool might have been due to the lack of passphrase as you seem to suggest. If you hadn't restarted your Freenas box in a while then the issue might have been there since day one and never saw it coming. I suspect you did an upgrade of Freenas and between 10 and 11 or so, the encryption algorithm has changed and also I believe due to the presence of a bug somewhere along the line.
With Coral many people have lost there encrypted pool due to such issue, myself included, but I had my pool replicated to several backups.
One of the issue with encryption was the inability of installing the redundant disk of a RAIDZ2 array. On a 5 disk RAIDZ2, the 2 dedundant disks would be made UN-available.

I believe in order to fix the issue I had to have passphrase set for the encrypted pool. Without a passphrase, the pool would be allowed to restore redundance, but only until the next reboot. The lack of passphrase would cause the algorithm to prevent restoring or updating the key and the pool would lost make the disk unavailble once more.

Without redundancy as you are, there is a chance you can loose your entire pool. Hence the reason why you are running on thin ice.
Also, this is not a position I like being put into, ( I mean me trying to giving you advice knowing what ever you will do can end up as a disaster.)


There re several scenario you should be made aware of in your actual situation. Rememebr you don't know what part of your data is backed up:

The more time you leave your system running in a degraded state with encrypted pool, the more chances of facing problems.

To that end here are the scenarion:

1) you leave the pool as is and you try to backup what you have. You can make things worse depending on your backup strategy, and it might take a while.

2) You can try replacing the disks advertised as unavailable. To do that you will have to follow the encrypted pool disk replacement. Be careful as not to add the disk to the pool. If you do you will be in deep troubles.
You should assign the 2 physical disk that are no longer in use (the one that were not able to be attached again.
If it lets you proceed witht the replacement and resilvering is underway, you will need to create a passphrase to the pool as resilvering is taking place. If you don't, the disk will be made unavailable at the end and you would have wasted valuable time.
You will need to save the GELI key and the Recovery GELI key to be safe.
I believe, once the passphrase has been created, it can be removed on the old GUI without too much hope of failing the recovery. Mind you I would at least keep it until the resilvering is complete and successful.

3) I have no more relevant solutions. Running a scrub on the degraded pool would be a waste of time.



Ok, Apollo - First of all - this is all either my fault, FreeNAS' fault, or both. This is not your fault, so don't feel like I can LOSE my data due to your advice. If I'm not comfortable with your advice, I don't have to do it! :)


That much aside, I'm not sure if we're on the same page here...

1) I restart the box all the time. It's not often that it's left on for more than 24 hours. It's mostly used to back up onto, &/or as a Plex server.

2) I had done no upgrade. It was a fresh installation of FreeNAS 11.1x.

3) From what you are saying, I think you are telling me that a 4 disk Z2 array can only lose 1 disk and still function. From everything I've seen, it can lose 2, and still be functional, although running in a degraded state. (Essentially, I would only actually need 2 functional disks to be able to access my data.)

4) I believe that my redundancy IS there, but I'm just locked out of it.

5) The pool is left as is - inaccessible, because I can't import/re-attach it at all for some reason. (All of it, including the error messages were in my first post here)

6) I DID try replacing the disk that was overheating. I shut the machine down, and swapped it out for another of the same drive, and powered it up and it couldn't access the drive, and now I can't access the pool at all, even though it sees it when I begin the "Import Volume" process. (See attachment)

Screen Shot 2019-05-22 at 23.23.42.png

7) I don't know if I'm misunderstanding you, or if you're misunderstanding me, but I have ZERO access to the pool/volume/whatever, right now. I can see that the disks are recognized by the system, if I select "View Disks" under "Storage", but there is simply no trace of the pool/volume at all, until I try to "Import Volume" - where it shows it's name, and then lets me select it, and then errors out.

8) Unless I don't know better, I can't generate a new passphrase for the pool/volume unless I already have access to it. I DO have the geli.key, but don't remember setting a passphrase, or if I did, what it was. I've tried my "normal" passwords that I use, and they don't seem to be doing it.

9) Scrubbing the pool can't be done, as the system cannot see the pool/volume.


I thank you for the time you took to respond to me, but I think that one (or both) of us isn't understanding the other. Hopefully I've cleared up any confusion I may have caused. If you have any questions for me to clarify or anything, just let me know!

Thanks again!
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Ok, Apollo - First of all - this is all either my fault, FreeNAS' fault, or both. This is not your fault, so don't feel like I can LOSE my data due to your advice. If I'm not comfortable with your advice, I don't have to do it! :)
All am I saying, is that this is not something to take likely.

That much aside, I'm not sure if we're on the same page here...

1) I restart the box all the time. It's not often that it's left on for more than 24 hours. It's mostly used to back up onto, &/or as a Plex server.
Then there is extra tear and wear due to power cycling.


2) I had done no upgrade. It was a fresh installation of FreeNAS 11.1x.
3) From what you are saying, I think you are telling me that a 4 disk Z2 array can only lose 1 disk and still function. From everything I've seen, it can lose 2, and still be functional, although running in a degraded state. (Essentially, I would only actually need 2 functional disks to be able to access my data.)
With 4 disk RAIDZ2, you can loose up to 2 disks. One more and your pool with be made unavailable. Could be fatal or not.
You just can't afford losing one more disk.

4) I believe that my redundancy IS there, but I'm just locked out of it.
You no longer have redundancy. If you add the disk back and resilver takes place, then redundancy will be restored on completion.

5) The pool is left as is - inaccessible, because I can't import/re-attach it at all for some reason. (All of it, including the error messages were in my first post here)

6) I DID try replacing the disk that was overheating. I shut the machine down, and swapped it out for another of the same drive, and powered it up and it couldn't access the drive, and now I can't access the pool at all, even though it sees it when I begin the "Import Volume" process. (See attachment)

View attachment 30877
When you import an encrypted pool, you HAVE to upload the Geli key, otherwise you will not be able to list the name of the pool.
Did you upload the Geli key then?

7) I don't know if I'm misunderstanding you, or if you're misunderstanding me, but I have ZERO access to the pool/volume/whatever, right now. I can see that the disks are recognized by the system, if I select "View Disks" under "Storage", but there is simply no trace of the pool/volume at all, until I try to "Import Volume" - where it shows it's name, and then lets me select it, and then errors out.
I can take a while for the import to take place (30 minute wouldn't be unexpected) before seeing the pool listed in GUI.
But from your point of view, Freenas is erroring out.

8) Unless I don't know better, I can't generate a new passphrase for the pool/volume unless I already have access to it. I DO have the geli.key, but don't remember setting a passphrase, or if I did, what it was. I've tried my "normal" passwords that I use, and they don't seem to be doing it.
If you get th name of the pool in the drop down list when GELI was uploaded but no passphrase was specified, this means your pool is not secured by a passphrase. If a passphrase was indeed present, then it would be required to enter it in order to get th name of the pool in the drop down list.
So it is clear your pool is not using any passphrase.

9) Scrubbing the pool can't be done, as the system cannot see the pool/volume.
Correct.

I thank you for the time you took to respond to me, but I think that one (or both) of us isn't understanding the other. Hopefully I've cleared up any confusion I may have caused. If you have any questions for me to clarify or anything, just let me know!

Thanks again!

Was this pool created under Freenas 11.1?
My understanding of the issue still revolves around the lack of a passphrase.
I also suspect the reason you can't import the 4 disk is that one of them contains the wrong encryption key.
Without a passphrase, the encryption doesn't hold it seems, when you try to rebuild a pool and the new disk will become unavailable.

With Freenas 11.X, there is a bug where a pool present in the system but is missing could cause import of volume to crash.
I would suggest you power off your system, remove the USB or SSD you are booting from and keep it safe.
I would temporarily disconnect the they other disks and simply install the same version of Freenas.
Potentially, if you have already been through one or more update on your original USB/SSD boot disk, then you could activate the previous boot environment. If the encrypted pool was already imported then, you would be able to plug your drives back and reboot. It should reconnect and mount the pool then.

If not, from the new Freenas boot disk, I would start from scratch and import the pool with the GELI key and no passphrase.
If you still get crashes, then I would try to narrow down which of the disk is causing the issue and exclude it from the import.
Repeat this stage until you have exhausted all solutions.

I suspect the crash maybe caused by a key clash.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
All am I saying, is that this is not something to take likely.

Ok, I just didn't want to come off sounding like a jerk. I absolutely understand it's something not to be taken lightly.

Then there is extra tear and wear due to power cycling.

& yes, I would agree with the extra wear and tear, however, the drives aren't loading, so there isn't much wearing or tearing going on

With 4 disk RAIDZ2, you can loose up to 2 disks. One more and your pool with be made unavailable. Could be fatal or not.
You just can't afford losing one more disk.

I partially agree with you. I say partially, because the data SHOULD still be on all of the disks. I believe the only issue here is unlocking/mounting any two of them, just long enough to copy the data off.

You no longer have redundancy. If you add the disk back and resilver takes place, then redundancy will be restored on completion.
When I said I had redundancy, I meant I had it in that the data (all the 1s and 0s) should still be on the disks. Again, only a matter of accessing them.

When you import an encrypted pool, you HAVE to upload the Geli key, otherwise you will not be able to list the name of the pool.
Did you upload the Geli key then?
Yes, I have a copy of, and did provide it when prompted during the volume import session.

I can take a while for the import to take place (30 minute wouldn't be unexpected) before seeing the pool listed in GUI.
But from your point of view, Freenas is erroring out.

I know it can take a long time - I'd be willing to wait days, if needed, but it simply gives me an error after under a minute. (The following screenshots are the order in which I see things happen)
Screen Shot 2019-05-21 at 18.04.51.png Screen Shot 2019-05-21 at 18.05.28.png Screen Shot 2019-05-21 at 18.05.56.png Screen Shot 2019-05-21 at 18.13.25.png


If you get th name of the pool in the drop down list when GELI was uploaded but no passphrase was specified, this means your pool is not secured by a passphrase. If a passphrase was indeed present, then it would be required to enter it in order to get th name of the pool in the drop down list.
So it is clear your pool is not using any passphrase.
Ok, good, then I am correct in thinking I did not use one. Glad to know that I was right.

Was this pool created under Freenas 11.1?
Yes, although I don't remember which version of 11.1. (I'm sorry, I have memory problems for medical reasons)

My understanding of the issue still revolves around the lack of a passphrase.
Well, if you are sure (from your earlier response) that this pool is not using a passphrase, but only the Geli.key, then the passphrase should not be a factor at all. Am I correct?

I also suspect the reason you can't import the 4 disk is that one of them contains the wrong encryption key. Without a passphrase, the encryption doesn't hold it seems, when you try to rebuild a pool and the new disk will become unavailable.

Ok, I think I understand what you're saying - however, why would it have worked for so long with the wrong passphrase? Maybe that's a question for the devs? (ABSOLUTELY no offense meant to you!)

With Freenas 11.X, there is a bug where a pool present in the system but is missing could cause import of volume to crash.

In that case, I am definitely reporting this as a bug. - ESPECIALLY because this just happened as I tried it again...
Screen Shot 2019-05-22 at 23.11.30.png Screen Shot 2019-05-22 at 23.11.39.png I can't find where any log would be that's more specific than "An error occurred!"

I would suggest you power off your system, remove the USB or SSD you are booting from and keep it safe.
I would temporarily disconnect the they other disks and simply install the same version of Freenas.

I have done that, and tried a fresh install & the problem persists. I'm considering trying this with all of the 11.1x versions of FreeNAS

Potentially, if you have already been through one or more update on your original USB/SSD boot disk, then you could activate the previous boot environment. If the encrypted pool was already imported then, you would be able to plug your drives back and reboot. It should reconnect and mount the pool then.

Ok, so I believe I tried that, now. If this is where you mean...
Screen Shot 2019-05-23 at 19.38.07.png

If not, from the new Freenas boot disk, I would start from scratch and import the pool with the GELI key and no passphrase. If you still get crashes, then I would try to narrow down which of the disk is causing the issue and exclude it from the import. Repeat this stage until you have exhausted all solutions.
I suspect the crash maybe caused by a key clash.

So, what MIGHT be actual PROGRESS.......
Restarting in the previous boot (which turned out to be Freenas 11.1-U6) and having only 2 drives magically has the volume appear in "Volumes"!!!

BUT......
It shows up as "Locked" "Locked" and "LOCKED"
Screen Shot 2019-05-23 at 20.02.08.png
I tried clicking the "Unlock" button at the bottom, and then it asks me "Do you want to unlock this volume?" and of course, I click "Yes", it thinks for a couple of seconds, and then says "Volume failed unlocked" for a few seconds, and then goes away.
Screen Shot 2019-05-23 at 20.06.41.png
It doesn't even ask for the key.

Thoughts?
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Hi Topher,

As you experienced, pool encryption will easily turn to a self-inflicted ransomware...

You said that you do not like the idea of doing some CLI. Unfortunately, I am pretty sure that you will have to do some here.

A first protection for you could be to do binary copy of your drives. You will need some extra drives and using any Unix operating system, you make a full binary copy of each of your good data drives to new backup drives (something like dd if=/dev/disk1 of=/dev/disk2 ; learn about DD if you need before trying to do this). Once you have binary copy of the drives, put them on the side and hope not to need them, but should you need, they will be there.

The error message asks you to check the pool status. So please, do so.

You already made it up to the point of seeing your pool's name and the volume, despite it is locked. So do it again and once mounted, open a command line and check the pool status. The command is zpool status, like this :

Code:
[root@atlas ~]# zpool status
  pool: JBCloud
 state: ONLINE
  scan: scrub repaired 0 in 1 days 12:32:23 with 0 errors on Tue May 14 12:32:28 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        JBCloud                                         ONLINE       0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/4dd5342d-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/5014f580-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/51e246e7-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/53bd9774-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/59388dec-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:13 with 0 errors on Fri May 10 03:47:13 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            da1p2   ONLINE       0     0     0

errors: No known data errors
[root@atlas ~]#


I think that is the next step and we will work from the output you will receive...

Good luck,
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Ok, so I believe I tried that, now. If this is where you mean...
View attachment 30923
Yes.
I would try all the boot solution and install all the drives.

So, what MIGHT be actual PROGRESS.......
Restarting in the previous boot (which turned out to be Freenas 11.1-U6) and having only 2 drives magically has the volume appear in "Volumes"!!!

BUT......
It shows up as "Locked" "Locked" and "LOCKED"
View attachment 30924
I tried clicking the "Unlock" button at the bottom, and then it asks me "Do you want to unlock this volume?" and of course, I click "Yes", it thinks for a couple of seconds, and then says "Volume failed unlocked" for a few seconds, and then goes away.
View attachment 30926
It doesn't even ask for the key.

Thoughts?
The key is not necessary. As I have stated, Freenas upgrade/update retains the state of the system (boot environment and various settings, such as keys, certificates and passwords...).
I still suspect you have within the same pool one or more of the disks with the wrong key.
Maybe it is something more serious. I think when you tried to remove the 2 parity disks, you might have caused some corruption to the pool. Without having the hardware at hands, I just can't think of where to go next. Not enough hard evidence.

I would do as Heracles suggest.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
Yes.
I would try all the boot solution and install all the drives.


The key is not necessary. As I have stated, Freenas upgrade/update retains the state of the system (boot environment and various settings, such as keys, certificates and passwords...).
I still suspect you have within the same pool one or more of the disks with the wrong key.
Maybe it is something more serious. I think when you tried to remove the 2 parity disks, you might have caused some corruption to the pool. Without having the hardware at hands, I just can't think of where to go next. Not enough hard evidence.

I would do as Heracles suggest.

Thank you for all of your help through this Apollo! I truly appreciate it!
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
Hi Topher,

As you experienced, pool encryption will easily turn to a self-inflicted ransomware...

You said that you do not like the idea of doing some CLI. Unfortunately, I am pretty sure that you will have to do some here.

A first protection for you could be to do binary copy of your drives. You will need some extra drives and using any Unix operating system, you make a full binary copy of each of your good data drives to new backup drives (something like dd if=/dev/disk1 of=/dev/disk2 ; learn about DD if you need before trying to do this). Once you have binary copy of the drives, put them on the side and hope not to need them, but should you need, they will be there.

The error message asks you to check the pool status. So please, do so.

You already made it up to the point of seeing your pool's name and the volume, despite it is locked. So do it again and once mounted, open a command line and check the pool status. The command is zpool status, like this :

Code:
[root@atlas ~]# zpool status
  pool: JBCloud
state: ONLINE
  scan: scrub repaired 0 in 1 days 12:32:23 with 0 errors on Tue May 14 12:32:28 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        JBCloud                                         ONLINE       0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/4dd5342d-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/5014f580-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/51e246e7-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/53bd9774-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0
            gptid/59388dec-fc77-11e8-8b4f-509a4c98555c  ONLINE       0     0 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:13 with 0 errors on Fri May 10 03:47:13 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            da1p2   ONLINE       0     0     0

errors: No known data errors
[root@atlas ~]#


I think that is the next step and we will work from the output you will receive...

Good luck,

It's more that I don't like trying something for the first time in a CLI that I'm not too familiar with, without a backup of my data.
Unfortunately, I don't have enough disk space laying around to dd a drive. So I'm basically going to just jump in with both feet! I will have 2 of the drives physically disconnected during this, though. Hopefully that will keep me at least partially safe.
So - here we go!

(Also, I did nothing to mount the volume/pool. It just showed up in the GUI. But still not in the CLI. It just shows it as "LOCKED" in the GUI. I'm pretty sure I posted a screenshot of it in the other thread.)

My results from zpool status are as follows:

Code:
[root@freenas ~]# zpool status                                                                                                     
  pool: freenas-boot                                                                                                               
 state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0 days 00:01:44 with 0 errors on Wed May 15 03:46:45 2019                                               
config:                                                                                                                             
                                                                                                                                    
        NAME        STATE     READ WRITE CKSUM                                                                                     
        freenas-boot  ONLINE       0     0     0                                                                                   
          da0p2     ONLINE       0     0     0                                                                                     
                                                                                                                                    
errors: No known data errors                                                                                                       
[root@freenas ~]#    


My biggest confusion is that I can now see it, in the GUI, but it says that it's locked. There's an unlock button, but it doesn't work. I have a key (The GELI.key that I generated), and it seems to work from one point of view, but not in this case. Because in this case, it doesn't even ask you for the key! lol! And there's a "Detach volume" button, but I don't want to detatch it again for fear of making this worse/repeat.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Something just stroke me about the "Locked" pool.
I have a few pools on my backup system I rotate. They are all encrypted. The one I pull out will show as "locked" even though the disk are no longer connected and as a result, I can't unlock them ( I don't detach the drives, I just shut the system down and then move the drives around).

So in simple words, Freenas doesn't regognize the disks assotiated with your pool as it was when you had Freenas 11.1-U6 installed.
I would still connect all the drives and proceed with a complete fresh install. You can also create a snapshot of the boot envirnemnt (same place you activated the previous boot) and on the new environment you can reset the web gui. It might work and this way you will not have to run a fresh install, but it is not without risks.
Do you have by any chance a Geli recovery key? If you do try it.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
they have said they've tried the geli key that they have already...
Not helping.

I know, but it is possible he forgot some GELI keys backed-up somewhere. Worth a try spending a bit of time to find out.
Beside I am not sure he has a GELI recovery key. The GELI key he has. Those are 2 distinct set of keys.
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
@Apollo @Heracles @artlessknave

So, I here's where I'm at...
1) I have connected ALL of the drives that have been used in the machine. So there are currently 4 internal 3TB drives, and 1 external 3TB drive connected via USB.

2) I have booted into the 11.1-U6 environment, gone to storage, and when "Volumes" comes up, I have an entirely new message:
Screen Shot 2019-05-24 at 18.48.29.png
Please note the difference in the "Used" & "Available" columns - AND - the fact that I have all of the buttons in the bottom left! (Detach volume, Scrub volume, Volume status... etc)

3) The "alert" light is flashing critical, giving me the following warning:
Screen Shot 2019-05-24 at 20.02.06.png

4) And when I click "View Disks" I get this (just for reference if you want - I don't think it's necessary)
Screen Shot 2019-05-24 at 20.34.33.png

.....thoughts?
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
And once done, what does zpool status tells you ?
 

TopherPSU

Dabbler
Joined
May 20, 2019
Messages
18
And once done, what does zpool status tells you ?
Code:
[root@freenas ~]# zpool status                                                                                                     
  pool: freenas-boot                                                                                                               
 state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0 days 00:01:44 with 0 errors on Wed May 15 03:46:45 2019                                               
config:                                                                                                                             
                                                                                                                                    
        NAME        STATE     READ WRITE CKSUM                                                                                     
        freenas-boot  ONLINE       0     0     0                                                                                   
          da1p2     ONLINE       0     0     0                                                                                     
                                                                                                                                    
errors: No known data errors                                                                                                       
[root@freenas ~]#   

& the console says this:
Code:
May 24 19:36:42 freenas syslog-ng[1651]: syslog-ng starting up; version='3.7.3'
May 24 19:36:42 freenas .
May 24 19:36:42 freenas Waiting (max 60 seconds) for system process `vnlru' to stop... done
May 24 19:36:42 freenas Waiting (max 60 seconds) for system process `syncer' to stop...
May 24 19:36:42 freenas Syncing disks, vnodes remaining... 0 0 0 done
May 24 19:36:42 freenas Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
May 24 19:36:42 freenas All buffers synced.
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap0.eli destroyed.
May 24 19:36:42 freenas GEOM_ELI: Detached mirror/swap0.eli on last close.
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap1.eli destroyed.
May 24 19:36:42 freenas GEOM_ELI: Detached mirror/swap1.eli on last close.
May 24 19:36:42 freenas Uptime: 37m54s
May 24 19:36:42 freenas GEOM_MIRROR: Device swap1: provider destroyed.
May 24 19:36:42 freenas GEOM_MIRROR: Device swap1 destroyed.
May 24 19:36:42 freenas GEOM_MIRROR: Device swap0: provider destroyed.
May 24 19:36:42 freenas GEOM_MIRROR: Device swap0 destroyed.
May 24 19:36:42 freenas Copyright (c) 1992-2017 The FreeBSD Project.
May 24 19:36:42 freenas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
May 24 19:36:42 freenas     The Regents of the University of California. All rights reserved.
May 24 19:36:42 freenas FreeBSD is a registered trademark of The FreeBSD Foundation.
May 24 19:36:42 freenas FreeBSD 11.1-STABLE #0 r321665+9902d126c39(freenas/11.1-stable): Tue Aug 21 12:24:37 EDT 2018
May 24 19:36:42 freenas root@nemesis.tn.ixsystems.com:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
May 24 19:36:42 freenas FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn)
May 24 19:36:42 freenas VT(efifb): resolution 800x600
May 24 19:36:42 freenas CPU: AMD E-450 APU with Radeon(tm) HD Graphics (1646.53-MHz K8-class CPU)
May 24 19:36:42 freenas Origin="AuthenticAMD"  Id=0x500f20  Family=0x14  Model=0x2  Stepping=0
May 24 19:36:42 freenas Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
May 24 19:36:42 freenas Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT>
May 24 19:36:42 freenas AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
May 24 19:36:42 freenas AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT>
May 24 19:36:42 freenas SVM: (disabled in BIOS) NP,NRIP,NAsids=8
May 24 19:36:42 freenas TSC: P-state invariant, performance statistics
May 24 19:36:42 freenas real memory  = 5351931904 (5104 MB)
May 24 19:36:42 freenas avail memory = 3658031104 (3488 MB)
May 24 19:36:42 freenas Event timer "LAPIC" quality 100
May 24 19:36:42 freenas ACPI APIC Table: <HPQOEM SLIC-CPC>
May 24 19:36:42 freenas FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
May 24 19:36:42 freenas FreeBSD/SMP: 1 package(s) x 2 core(s)
May 24 19:36:42 freenas WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
May 24 19:36:42 freenas ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170728/tbfadt-796)
May 24 19:36:42 freenas ioapic0 <Version 2.1> irqs 0-23 on motherboard
May 24 19:36:42 freenas SMP: AP CPU #1 Launched!
May 24 19:36:42 freenas Timecounter "TSC" frequency 1646529393 Hz quality 800
May 24 19:36:42 freenas random: entropy device external interface
May 24 19:36:42 freenas kbd1 at kbdmux0
May 24 19:36:42 freenas module_register_init: MOD_LOAD (vesa, 0xffffffff80fc84c0, 0) error 19
May 24 19:36:42 freenas nexus0
May 24 19:36:42 freenas cryptosoft0: <software crypto> on motherboard
May 24 19:36:42 freenas aesni0: No AESNI support.
May 24 19:36:42 freenas padlock0: No ACE support.
May 24 19:36:42 freenas acpi0: <HPQOEM SLIC-CPC> on motherboard
May 24 19:36:42 freenas acpi0: Power Button (fixed)
May 24 19:36:42 freenas cpu0: <ACPI CPU> on acpi0
May 24 19:36:42 freenas cpu1: <ACPI CPU> on acpi0
May 24 19:36:42 freenas attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
May 24 19:36:42 freenas Timecounter "i8254" frequency 1193182 Hz quality 0
May 24 19:36:42 freenas Event timer "i8254" frequency 1193182 Hz quality 100
May 24 19:36:42 freenas atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
May 24 19:36:42 freenas atrtc0: registered as a time-of-day clock, resolution 1.000000s
May 24 19:36:42 freenas Event timer "RTC" frequency 32768 Hz quality 0
May 24 19:36:42 freenas hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
May 24 19:36:42 freenas Timecounter "HPET" frequency 14318180 Hz quality 950
May 24 19:36:42 freenas Event timer "HPET" frequency 14318180 Hz quality 550
May 24 19:36:42 freenas Event timer "HPET1" frequency 14318180 Hz quality 450
May 24 19:36:42 freenas Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
May 24 19:36:42 freenas acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
May 24 19:36:42 freenas pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
May 24 19:36:42 freenas pci0: <ACPI PCI bus> on pcib0
May 24 19:36:42 freenas vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xfeb00000-0xfeb3ffff irq 18 at device 1.0 on pci0
May 24 19:36:42 freenas vgapci0: Boot video device
May 24 19:36:42 freenas pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
May 24 19:36:42 freenas pcib1: [GIANT-LOCKED]
May 24 19:36:42 freenas ahci0: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xfeb4b000-0xfeb4b3ff irq 19 at device 17.0 on pci0
May 24 19:36:42 freenas ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
May 24 19:36:42 freenas ahci0: quirks=0x22000<ATI_PMP_BUG,1MSI>
May 24 19:36:42 freenas ahcich0: <AHCI channel> at channel 0 on ahci0
May 24 19:36:42 freenas ahcich1: <AHCI channel> at channel 1 on ahci0
May 24 19:36:42 freenas ahcich2: <AHCI channel> at channel 2 on ahci0
May 24 19:36:42 freenas ahcich3: <AHCI channel> at channel 3 on ahci0
May 24 19:36:42 freenas ahcich4: <AHCI channel> at channel 4 on ahci0
May 24 19:36:42 freenas ahcich5: <AHCI channel> at channel 5 on ahci0
May 24 19:36:42 freenas ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfeb4a000-0xfeb4afff irq 18 at device 18.0 on pci0
May 24 19:36:42 freenas usbus0 on ohci0
May 24 19:36:42 freenas usbus0: 12Mbps Full Speed USB v1.0
May 24 19:36:42 freenas ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfeb49000-0xfeb490ff irq 17 at device 18.2 on pci0
May 24 19:36:42 freenas usbus1: EHCI version 1.0
May 24 19:36:42 freenas usbus1 on ehci0
May 24 19:36:42 freenas usbus1: 480Mbps High Speed USB v2.0
May 24 19:36:42 freenas ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfeb48000-0xfeb48fff irq 18 at device 19.0 on pci0
May 24 19:36:42 freenas usbus2 on ohci1
May 24 19:36:42 freenas usbus2: 12Mbps Full Speed USB v1.0
May 24 19:36:42 freenas ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfeb47000-0xfeb470ff irq 17 at device 19.2 on pci0
May 24 19:36:42 freenas usbus3: EHCI version 1.0
May 24 19:36:42 freenas usbus3 on ehci1
May 24 19:36:42 freenas usbus3: 480Mbps High Speed USB v2.0
May 24 19:36:42 freenas pci0: <multimedia, HDA> at device 20.2 (no driver attached)
May 24 19:36:42 freenas isab0: <PCI-ISA bridge> at device 20.3 on pci0
May 24 19:36:42 freenas isa0: <ISA bus> on isab0
May 24 19:36:42 freenas pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
May 24 19:36:42 freenas pci1: <ACPI PCI bus> on pcib2
May 24 19:36:42 freenas ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfeb46000-0xfeb46fff irq 18 at device 20.5 on pci0
May 24 19:36:42 freenas usbus4 on ohci2
May 24 19:36:42 freenas usbus4: 12Mbps Full Speed USB v1.0
May 24 19:36:42 freenas pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
May 24 19:36:42 freenas pci2: <ACPI PCI bus> on pcib3
May 24 19:36:42 freenas pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
May 24 19:36:42 freenas pci3: <ACPI PCI bus> on pcib4
May 24 19:36:42 freenas re0: <RealTek 810xE PCIe 10/100baseTX> port 0xe000-0xe0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 17 at device 0.0 on pci3
May 24 19:36:42 freenas re0: Using 1 MSI-X message
May 24 19:36:42 freenas re0: turning off MSI enable bit.
May 24 19:36:42 freenas re0: Chip rev. 0x40800000
May 24 19:36:42 freenas re0: MAC rev. 0x00200000
May 24 19:36:42 freenas miibus0: <MII bus> on re0
May 24 19:36:42 freenas rlphy0: <RTL8201E 10/100 media interface> PHY 1 on miibus0
May 24 19:36:42 freenas rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
May 24 19:36:42 freenas re0: Using defaults for TSO: 65518/35/2048
May 24 19:36:42 freenas re0: Ethernet address: 38:60:77:0e:8b:b4
May 24 19:36:42 freenas ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfeb45000-0xfeb45fff irq 18 at device 22.0 on pci0
May 24 19:36:42 freenas usbus5 on ohci3
May 24 19:36:42 freenas usbus5: 12Mbps Full Speed USB v1.0
May 24 19:36:42 freenas ehci2: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfeb44000-0xfeb440ff irq 17 at device 22.2 on pci0
May 24 19:36:42 freenas usbus6: EHCI version 1.0
May 24 19:36:42 freenas usbus6 on ehci2
May 24 19:36:42 freenas usbus6: 480Mbps High Speed USB v2.0
May 24 19:36:42 freenas amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
May 24 19:36:42 freenas acpi_button0: <Power Button> on acpi0
May 24 19:36:42 freenas orm0: <ISA Option ROM> at iomem 0xce800-0xcf7ff on isa0
May 24 19:36:42 freenas amdsbwd0: <AMD SB8xx/SB9xx/Axx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0
May 24 19:36:42 freenas atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
May 24 19:36:42 freenas atkbd0: <AT Keyboard> irq 1 on atkbdc0
May 24 19:36:42 freenas kbd0 at atkbd0
May 24 19:36:42 freenas atkbd0: [GIANT-LOCKED]
May 24 19:36:42 freenas hwpstate0: <Cool`n'Quiet 2.0> on cpu0
May 24 19:36:42 freenas ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
May 24 19:36:42 freenas to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
May 24 19:36:42 freenas ZFS filesystem version: 5
May 24 19:36:42 freenas ZFS storage pool version: features support (5000)
May 24 19:36:42 freenas Timecounters tick every 1.000 msec
May 24 19:36:42 freenas freenas_sysctl: adding account.
May 24 19:36:42 freenas freenas_sysctl: adding directoryservice.
May 24 19:36:42 freenas freenas_sysctl: adding middlewared.
May 24 19:36:42 freenas freenas_sysctl: adding network.
May 24 19:36:42 freenas freenas_sysctl: adding services.
May 24 19:36:42 freenas ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
May 24 19:36:42 freenas ugen0.1: <ATI OHCI root HUB> at usbus0
May 24 19:36:42 freenas ugen6.1: <ATI EHCI root HUB> at usbus6
May 24 19:36:42 freenas uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
May 24 19:36:42 freenas uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
May 24 19:36:42 freenas ugen3.1: <ATI EHCI root HUB> at usbus3
May 24 19:36:42 freenas ugen4.1: <ATI OHCI root HUB> at usbus4
May 24 19:36:42 freenas uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
May 24 19:36:42 freenas uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
May 24 19:36:42 freenas ugen1.1: <ATI EHCI root HUB> at usbus1
May 24 19:36:42 freenas ugen5.1: <ATI OHCI root HUB> at usbus5
May 24 19:36:42 freenas uhub4: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
May 24 19:36:42 freenas uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
May 24 19:36:42 freenas ugen2.1: <ATI OHCI root HUB> at usbus2
May 24 19:36:42 freenas uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
May 24 19:36:42 freenas uhub3: 2 ports with 2 removable, self powered
May 24 19:36:42 freenas uhub0: 5 ports with 5 removable, self powered
May 24 19:36:42 freenas uhub5: 4 ports with 4 removable, self powered
May 24 19:36:42 freenas uhub6: 5 ports with 5 removable, self powered
May 24 19:36:42 freenas uhub1: 4 ports with 4 removable, self powered
May 24 19:36:42 freenas uhub2: 5 ports with 5 removable, self powered
May 24 19:36:42 freenas uhub4: 5 ports with 5 removable, self powered
May 24 19:36:42 freenas ugen3.2: <vendor 0x1a40 USB 2.0 Hub> at usbus3
May 24 19:36:42 freenas uhub7 on uhub2
May 24 19:36:42 freenas uhub7: <vendor 0x1a40 USB 2.0 Hub, class 9/0, rev 2.00/1.11, addr 2> on usbus3
May 24 19:36:42 freenas ugen0.2: <Dell Dell USB Keyboard> at usbus0
May 24 19:36:42 freenas ukbd0 on uhub0
May 24 19:36:42 freenas ukbd0: <EP1 Interrupt> on usbus0
May 24 19:36:42 freenas kbd2 at ukbd0
May 24 19:36:42 freenas uhub7: 4 ports with 4 removable, self powered
May 24 19:36:42 freenas ugen3.3: <JMicron USB to ATAATAPI bridge> at usbus3
May 24 19:36:42 freenas umass0 on uhub7
May 24 19:36:42 freenas umass0: <MSC Bulk-Only Transfer> on usbus3
May 24 19:36:42 freenas umass0:  SCSI over Bulk-Only; quirks = 0x4000
May 24 19:36:42 freenas umass0:7:0: Attached to scbus7
May 24 19:36:42 freenas ugen1.2: <SanDisk Cruzer Fit> at usbus1
May 24 19:36:42 freenas umass1 on uhub4
May 24 19:36:42 freenas umass1: <SanDisk Cruzer Fit, class 0/0, rev 2.10/1.00, addr 2> on usbus1
May 24 19:36:42 freenas umass1:  SCSI over Bulk-Only; quirks = 0x8100
May 24 19:36:42 freenas (probe0:umass-sim0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
May 24 19:36:42 freenas umass1:8:1: Attached to scbus8
May 24 19:36:42 freenas (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
May 24 19:36:42 freenas (probe0:umass-sim0:0:0:0): SCSI status: Check Condition
May 24 19:36:42 freenas (probe0:umass-sim0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
May 24 19:36:42 freenas (probe0:umass-sim0:0:0:0): Error 22, Unretryable error
May 24 19:36:42 freenas ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
May 24 19:36:42 freenas ada0: <ST33000651NS G009> ATA8-ACS SATA 3.x device
May 24 19:36:42 freenas ada0: Serial Number Z294CBSQ
May 24 19:36:42 freenas ada0: 300.000MB/s transfersda0 at umass-sim0 bus 0 scbus7 target 0 lun 0
May 24 19:36:42 freenas (da0: SATA 2.x, <ST330006 51NS > Fixed Direct Access SPC-3 SCSI device
May 24 19:36:42 freenas UDMA6, PIO 8192bytes)
May 24 19:36:42 freenas da0: Serial Number 000000000000
May 24 19:36:42 freenas ada0: Command Queueing enabled
May 24 19:36:42 freenas da0: 40.000MB/s transfersada0: 2861588MB (5860533168 512 byte sectors)
May 24 19:36:42 freenas da0: 2861588MB (5860533168 512 byte sectors)
May 24 19:36:42 freenas ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
May 24 19:36:42 freenas da0: quirks=0x2<NO_6_BYTE>
May 24 19:36:42 freenas ada1: <ST33000651NS G008> ATA8-ACS SATA 3.x device
May 24 19:36:42 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
May 24 19:36:42 freenas ada1: Serial Number Z293BX3V
May 24 19:36:42 freenas da1: ada1: 300.000MB/s transfers<SanDisk Cruzer Fit 1.00> Removable Direct Access SPC-4 SCSI device
May 24 19:36:42 freenas (SATA 2.x, da1: Serial Number 4C530001100526102492
May 24 19:36:42 freenas UDMA6, PIO 8192bytesda1: 40.000MB/s transfers)
May 24 19:36:42 freenas ada1: Command Queueing enabled
May 24 19:36:42 freenas ada1: 2861588MB (5860533168 512 byte sectors)
May 24 19:36:42 freenas da1: 7632MB (15630336 512 byte sectors)
May 24 19:36:42 freenas ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
May 24 19:36:42 freenas da1: quirks=0x2<NO_6_BYTE>
May 24 19:36:42 freenas ada2: <ST33000651NS G008> ATA8-ACS SATA 3.x device
May 24 19:36:42 freenas ada2: Serial Number Z293BQB2
May 24 19:36:42 freenas ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May 24 19:36:42 freenas ada2: Command Queueing enabled
May 24 19:36:42 freenas ada2: 2861588MB (5860533168 512 byte sectors)
May 24 19:36:42 freenas ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
May 24 19:36:42 freenas ada3: <ST3000DM001-1ER166 CC43> ACS-2 ATA SATA 3.x device
May 24 19:36:42 freenas ada3: Serial Number Z5006JTR
May 24 19:36:42 freenas ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May 24 19:36:42 freenas ada3: Command Queueing enabled
May 24 19:36:42 freenas ada3: 2861588MB (5860533168 512 byte sectors)
May 24 19:36:42 freenas ada3: quirks=0x1<4K>
May 24 19:36:42 freenas random: unblocking device.
May 24 19:36:42 freenas Trying to mount root from zfs:freenas-boot/ROOT/default []...
May 24 19:36:42 freenas kernel: re0: link state changed to UP
May 24 19:36:42 freenas kernel: re0: link state changed to UP
May 24 19:36:42 freenas SVM: disabled by BIOS.
May 24 19:36:42 freenas module_register_init: MOD_LOAD (vmm, 0xffffffff829f3400, 0) error 6
May 24 19:36:42 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
May 24 19:36:42 freenas GEOM_MIRROR: Device mirror/swap0 launched (2/2).
May 24 19:36:42 freenas GEOM_MIRROR: Device mirror/swap1 launched (2/2).
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap0.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 128
May 24 19:36:42 freenas GEOM_ELI:     Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap1.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 128
May 24 19:36:42 freenas GEOM_ELI:     Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/bda6d122-9131-11e8-991f-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI:     Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/bec972c2-9131-11e8-991f-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI:     Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/008eab85-d6e2-11e8-80d6-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI:     Crypto: software
May 24 19:36:42 freenas hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> K8/4/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
May 24 19:36:42 freenas kernel: re0: link state changed to DOWN
May 24 19:36:42 freenas kernel: re0: link state changed to DOWN
May 24 19:36:42 freenas kernel: re0: link state changed to UP
May 24 19:36:42 freenas kernel: re0: link state changed to UP
May 24 19:36:47 freenas nfsd: can't register svc name
May 24 19:36:47 freenas ntpd[2077]: ntpd 4.2.8p10-a (1): Starting
May 24 19:36:50 freenas proftpd[2226]: 127.0.0.1 - ProFTPD 1.3.6 (stable) (built Tue Aug 21 2018 16:56:03 UTC) standalone mode STARTUP
May 24 19:37:14 freenas smartd[2618]: Device: /dev/ada3, 16 Currently unreadable (pending) sectors
May 24 19:37:15 freenas smartd[2618]: Device: /dev/ada3, 16 Offline uncorrectable sectors
May 24 19:37:16 freenas root: /etc/rc: WARNING: failed precmd routine for minio
May 24 20:07:17 freenas smartd[2618]: Device: /dev/ada3, 16 Currently unreadable (pending) sectors
May 24 20:07:17 freenas smartd[2618]: Device: /dev/ada3, 16 Offline uncorrectable sectors
May 24 20:37:16 freenas smartd[2618]: Device: /dev/ada3, 16 Currently unreadable (pending) sectors
May 24 20:37:16 freenas smartd[2618]: Device: /dev/ada3, 16 Offline uncorrectable sectors
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Hi again Topher,

I reviewed the results and unfortunately, I can not help you more... I never touched pool encryption because it is both way too dangerous and basically useless. Almost all of my data is encrypted in my NAS, but not at the pool level. I host my private Cloud based on Nextcloud and I use Nextcloud's server side encryption. I did a lot of tests, backup and restore and ensured myself I could recover my Dev server before deploying it to my prod server. I experimented with encryption in Dev for 1 year before putting it on the production side.

Is your pool recoverable from here ? Personnaly, I do not know. It may be, because the boot mostly complains about ada3, so a single problematic drive should be Ok, but because of encryption, I am not sure. It is enough to screw a single crypto key to skrew up an entire cryptogram, no matter it is a file, a drive or a pool.

Let see if other people here have more experience with pool encryption, but as for me, I always knew enough about it for not touching it...

Good luck,
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
@Apollo @Heracles @artlessknave

So, I here's where I'm at...Reply
1) I have connected ALL of the drives that have been used in the machine. So there are currently 4 internal 3TB drives, and 1 external 3TB drive connected via USB.

2) I have booted into the 11.1-U6 environment, gone to storage, and when "Volumes" comes up, I have an entirely new message:
View attachment 30938
Please note the difference in the "Used" & "Available" columns - AND - the fact that I have all of the buttons in the bottom left! (Detach volume, Scrub volume, Volume status... etc)
Here is a screenshot of my backup server.
Both pools are not currently connected to the server:
HGST-RAIDZ1-2 ( I think this one is not encrypted. I will have to check but I don't have access to my server at te moment.
HGST-RAID1 has encryption but no passphrase I think.

All encrypted but missing pools will only show the "Detach Volume" and "Unlock" icons.

In your case, the GUI shows the icons for a pool that is indeed encrypted and unlocked, however the Status shown as "Locked" is perplexing.
This is beyond my experience, but my gut's feeling suggest you may have an extra level of encryption within the pool itself.

3) The "alert" light is flashing critical, giving me the following warning:
View attachment 30939
This is expected.
4) And when I click "View Disks" I get this (just for reference if you want - I don't think it's necessary)
View attachment 30940
This means your hardware and Freenas see the physical drives. This is expected and this also means the disk are connected properly. This doesn't imply Freenas have access to their content, though.


.....thoughts?
From your message output:

May 24 19:36:42 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
May 24 19:36:42 freenas GEOM_MIRROR: Device mirror/swap0 launched (2/2).
May 24 19:36:42 freenas GEOM_MIRROR: Device mirror/swap1 launched (2/2).
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap0.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 128
May 24 19:36:42 freenas GEOM_ELI: Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device mirror/swap1.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 128
May 24 19:36:42 freenas GEOM_ELI: Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/bda6d122-9131-11e8-991f-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI: Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/bec972c2-9131-11e8-991f-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI: Crypto: software
May 24 19:36:42 freenas GEOM_ELI: Device gptid/008eab85-d6e2-11e8-80d6-3860770e8bb4.eli created.
May 24 19:36:42 freenas GEOM_ELI: Encryption: AES-XTS 256
May 24 19:36:42 freenas GEOM_ELI: Crypto: software

This suggest that decryption of the pool is successful up to this point , but only 3 disks are used for the pool. Are you sure you have a 4 disk RAIDZ2? and not RAIDZ1? I don't know if the missing disk would trigger an error on the decryption. I think it would say it can't find one of the disk.

Do you have more of that output messages showing information after what you have listed?
I don't know if errors have occurred after that.
 

Attachments

  • Freenas_encrypted_pools.png
    Freenas_encrypted_pools.png
    8.8 KB · Views: 333
Top