Help please! Error decrypting pool Freenas 11_3 beta

Zennie12

Cadet
Joined
Dec 5, 2019
Messages
1
I have a huge problem with decrypting my hdd-pool. I get the following error while uploading the key file. I dont understand what this means and how I can solve it.

FAILED
[EFAULT] Pool could not be imported: 6 devices failed to decrypt.

more info:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 95, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 51, in _run
return await self._call(name, serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 953, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
zfs.import_pool(found, found.name, options, any_host=any_host)
File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
zfs.import_pool(found, found.name, options, any_host=any_host)
File "libzfs.pyx", line 870, in libzfs.ZFS.import_pool
libzfs.ZFSException: I/O error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1656, in unlock
'cachefile': ZPOOL_CACHE_FILE,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1074, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1094, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1029, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1003, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('I/O error',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
await self.future
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 949, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1668, in unlock
raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 6 devices failed to decrypt.
 
Joined
Oct 18, 2018
Messages
969
Hi @Zennie12. Full disclaimer, I'm not 100% sure what changes have happened between 11.2 and 11.3 WRT to encryption, so I'll tread lightly.

I will suggest that possibly the following thing may have happened to you.

When you create an encrypted pool it gives you the opportunity to download a key and save it at pool creation time. This is often saved as geli.key. Then, you may add a recovery key and download that one, I'll call it recovery.key. You may choose to also add a passphrase. The passphrase ONLY applies to geli.key.

When you try to import an encrypted pool and you used a passphrase originally you should provide either
geli.key AND the passphrase
OR
recovery.key WITHOUT the passphrase

Give that a try. Do note that if you use the recovery.key you likely will want to rekey your pool, readd your passphrase, and redownload the recovery key. You can reuse the passphrase if you like but you will get new keys in this process and should be sure to secure those and not mix them up with the old ones.

If you're able to unlock the pool and REALLY want to use the same keys you can check out my post here. I haven't tested it on 11.3 yet though so use at your own risk.

Also, just a side note. Would you mind editing your above post and add code tags around your code above? You do it like this

[CODE]
some code here
[/CODE]

so that it looks like this
Code:
some code here
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
Did you ever find a solution?? PLEASE say you did.

I have an encrypted pool + passphrase. Normally on reboot I just enter the passphrase and the pool unlocks. For some reason now when I enter the passphrase I get the following error:

FAILED
[EFAULT] Pool could not be imported: 3 devices failed to decrypt.
remove_circle_outlineMore info...


Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 95, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 51, in _run
return await self._call(name, serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
zfs.import_pool(found, found.name, options, any_host=any_host)
File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 380, in import_pool
raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 49485231544439643 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1656, in unlock
'cachefile': ZPOOL_CACHE_FILE,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1074, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1094, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1029, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1003, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 49485231544439643 not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
await self.future
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1668, in unlock
raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 3 devices failed to decrypt.

I did update to 11.3-RELEASE on 1/28/2020. I do have the following keys saved from 1/29/2020:

pool_eightTB.spinners_encryption.key
pool_fourTB.spinners_encryption.key
pool_tenTB.spinners_encryption.key

I did delete the 'fourTB.spinners' pool and wiped the data prior to reboot. This would not have cause the problem, right?

-Dave
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
If you did wipe the data for 'fourTB.spinners', then everything you had on that pool is now gone.
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
Yeah... I know... but I am sure I didn't wipe

At the end of the that first paragraph in my log it says "middlewared.service_exception.CallError: [ENOENT] Pool 49485231544439643 not found."
The fact that it says 'not found' is giving me some hope (that line is not in the original log posted by Zennie12).

I tried to export/disconnect the 'pool_eightTB.spinners' volume and then re-attach. The problem no disks populate in the dropdown:

2020.02.08.at.21.33.05.ScreenShot.from.RYZEN-2700X.png

Is there a command that can search the disks for password+encrypted volumes?

I tried 'zpool list -v' but it only shows my boot mirror

Here is mine...
root@freenas[~]# zpool status -v
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:27 with 0 errors on Fri Feb 7 03:46:29 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da22p2 ONLINE 0 0 0
da23p2 ONLINE 0 0 0

errors: No known data errors

Any help appreciated

-Dave
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
You did say you wiped the 'fourTB.spinners' pool, isn't what you did?
I am myself trying to recover one of my encrypted volume and I am getting a few errors on import through the GUI. I don't have access to my latest backup as the system is remote, but decryption seems to work at some level.
Even though the volume doesn't appear in the GUI, a "zpool status" under CLI shows the pool present and listing of the snapshot is also possible.
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
The pool "fourTB.spinners" yes... I wiped... but that was intentional. Preceding this mess I was trying to set the seven 4tb drives I have as a striped mirror (+ one in reserve). I was having a little difficulty because one of the drives was just a tad smaller than the others... but that is a whole other story. Somewhere along the line I thought it might be a good idea for a reboot. Really am regretting that decision.

I never wiped the two other pools: tenTB.spinners or eightTB.spinners

my 'tenTB.spinners' is still sitting in the web ui:
2020.02.08.at.23.39.19.ScreenShot.from.RYZEN-2700X.png


my 'eightTB.spinners' is not there because earlier today I tried to disconnect/reconnect. The problem is no disk populate in the drop-down
2020.02.08.at.23.41.21.ScreenShot.from.RYZEN-2700X.png

Getting late now so going to pick this up tomorrow.

Trouble started at 11:38 this morning (Feb 8th) when I triggered the reboot. I will be digging through the attached logs tomorrow.
If you know of any other for me to look at please let me know.

I appreciate your help.

-Dave

PS: Maybe the most frustrating thing is I was having a similar issue when I upgraded to version 11. I was only able to unlock my pools by going into classic webui... but I read somewhere the classic interface is gone with 11.3.
 

Attachments

  • console+daemon+debug+middlewared.logs.zip
    108.1 KB · Views: 291
Joined
Oct 18, 2018
Messages
969
my 'eightTB.spinners' is not there because earlier today I tried to disconnect/reconnect. The problem is no disk populate in the drop-down
What do you get when you do camcontrol devlist? Are you 100% certain you have all of the data and power plugs fully and properly plugged in? Also, what do you see under Storage->Disks?
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
camcontrol devlist give me the following:
root@freenas[~]# camcontrol devlist
<ATA ST10000NM0086-2A SN05> at scbus0 target 9 lun 0 (pass0,da0)
<ATA ST10000NM0086-2A SN05> at scbus0 target 10 lun 0 (pass1,da1)
<ATA ST10000NM0086-2A SN05> at scbus0 target 11 lun 0 (pass2,da2)
<ATA ST10000NM0086-2A SN05> at scbus0 target 12 lun 0 (pass3,da3)
<ATA ST10000NM0086-2A SN05> at scbus0 target 13 lun 0 (pass4,da4)
<ATA ST10000NM0086-2A SN05> at scbus0 target 14 lun 0 (pass5,da5)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 18 lun 0 (pass6,da6)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 19 lun 0 (pass7,da7)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 20 lun 0 (pass8,da8)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 21 lun 0 (pass9,da9)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 22 lun 0 (pass10,da10)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 23 lun 0 (pass11,da11)
<ATA ST4000VN000-1H41 SC43> at scbus1 target 9 lun 0 (pass12,da12)
<DELL MD1200 1.01> at scbus1 target 10 lun 0 (pass13,ses0)
<ATA WDC WD40EFRX-68W 0A82> at scbus1 target 11 lun 0 (pass14,da13)
<ATA WDC WD40EFRX-68W 0A82> at scbus1 target 12 lun 0 (pass15,da14)
<ATA ST4000VN000-1H41 SC44> at scbus1 target 13 lun 0 (pass16,da15)
<ATA ST4000DM000-1F21 CC52> at scbus1 target 14 lun 0 (pass17,da16)
<ATA ST4000DM000-1F21 CC51> at scbus1 target 15 lun 0 (pass18,da17)
<ATA WDC WD40EFRX-68W 0A82> at scbus1 target 16 lun 0 (pass19,da18)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 17 lun 0 (pass20,da19)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 18 lun 0 (pass21,da20)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 19 lun 0 (pass22,da21)
<SanDisk Ultra Fit 1.00> at scbus3 target 0 lun 0 (pass23,da22)
<PNY USB 3.0 FD 1100> at scbus4 target 0 lun 0 (pass24,da23)
root@freenas[~]#

& all the disks show up in storage -> disks

2020.02.09.at.09.47.35.ScreenShot.from.RYZEN-2700X.png

Machine is a Dell r720xd with redundant power, jbod controller (LSI), ecc memory... the one weak link is that the OS is on mirrored usb sticks. I planned on upgrading to SSD but always figured if they went I would still be able to recover the data on the drives.

Does anyone know if 'zpool status -v' is supposed to show show encrypted+password pools?

-Dave
 
Joined
Oct 18, 2018
Messages
969
Does anyone know if 'zpool status -v' is supposed to show show encrypted+password pools?
zpool status will not show pools not currently known to the system. An exported and locked pool will not be available. To see importable pools you would use zpool import but again, if the drives are locked zfs has no knowledge of the pool.

In that list above, do you see the disks for eightTB.spinners? If you do, and those disks do not show up under the disks tab in the GUI during import it may be worth filing a bug. You can verify your pool yourself by trying to manually unlock the pool using geli attach -k <key> /dev/gptid/<device>. If you're able to unlock all of your drives for eightTB.spinners then give zpool import a shot to see if zfs sees your pool. If it does, you're in luck and it might be a UI bug. If not, you may have the wrong key, the wrong disks, etc. I do not recommend you import the pool via the CLI. FreeNAS requires additional configuration etc to manage pools which will not be set up if imported via the CLI. The goal of the above steps is to confirm that your keys are correct and that your pool is still there.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
@theyost, I was having the same issue as you when trying to import an encrypted pool and selecting the list of disk would not be populated.
zpool status will not show pools not currently known to the system. An exported and locked pool will not be available. To see importable pools you would use zpool import but again, if the drives are locked zfs has no knowledge of the pool.
Incorrect for Freenas 11.3 Release. When I try to import the encrypted pool from the GUI, it fails and as a result the pool will not be visible in the GUI. However, I can run "zpool list" and "zfs list -t snapshot ..." and I can see the content of the volume.

I have noticed some error messages showing up in CLI after the disk are being decrypted with mention about "consumer destroyed" or something like that.
I think, because ZFS is aware of the mounted volume but GUI isn't is the reason why the disk do not show up in the list during volume import.

In that list above, do you see the disks for eightTB.spinners? If you do, and those disks do not show up under the disks tab in the GUI during import it may be worth filing a bug. You can verify your pool yourself by trying to manually unlock the pool using geli attach -k <key> /dev/gptid/<device>. If you're able to unlock all of your drives for eightTB.spinners then give zpool import a shot to see if zfs sees your pool. If it does, you're in luck and it might be a UI bug. If not, you may have the wrong key, the wrong disks, etc. I do not recommend you import the pool via the CLI. FreeNAS requires additional configuration etc to manage pools which will not be set up if imported via the CLI. The goal of the above steps is to confirm that your keys are correct and that your pool is still there.
If the wrong key is used, then the pool would not be listed.
I think there is a bug with the encryption mechanism and every new release has this weakness. Something always break.
 
Joined
Oct 18, 2018
Messages
969
Incorrect for Freenas 11.3 Release. When I try to import the encrypted pool from the GUI, it fails and as a result the pool will not be visible in the GUI.
I think there may be some confusion. Perhaps I am misunderstanding the state of our system. If the system, zfs, does not know about a pool, zpool status will not display the pool. I just tested this in 11.3 RELEASE by creating pool, exporting it, and leaving the pool connected. zpool status did not show the pool but zpool import did. I attempted the same thing with an encrypted pool to the same effect. If your pool is not imported zpool status will not show the pool. If your disks are locked zpool import will not list it.

This point is important because the state of the pool with respect to what zfs knows about will help determine where the issue is.

Can you clarify and post what you get with both zpool status and zpool import, which pool is the one in question, and whether you expect it to be locked, imported, etc?

However, I can run "zpool list" and "zfs list -t snapshot ..." and I can see the content of the volume.
This suggests to me that your pool is unlocked already and imported as far as zfs is concerned. This might explain why the list of disks to select is unavailable. If this is the case perhaps the GUI has somehow messed up such that the pool did not get properly configured for the GUI? What do you get when you check the database? sqlite3 /data/freenas-v1.db 'select * from storage_volume;'.
edit: corrected typo
If the wrong key is used, then the pool would not be listed.
I think there is a bug with the encryption mechanism and every new release has this weakness. Something always break.
I agree, I think this is possibly a bug and potentially worth filing. If it is a bug filing it will help ensure it gets fixed.
 
Last edited:

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
@PhiloEpisteme:
Content of GUI:
zfs-GUI.png
zfs_info_scrub.png


Result of "zpool status":

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@freenas-E3[/]# zpool status
pool: SG-RAIDZ2-1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0 days 22:19:36 with 0 errors on Sat Jan 25 16:13:08 2020
config:

NAME STATE READ WRITE CKSUM
SG-RAIDZ2-1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/b1a45c54-c90b-11e8-8648-ac1f6b251a24.eli ONLINE 0 0 0
gptid/b89fb6f4-ea9a-11e6-b10c-001d091e503b.eli ONLINE 0 0 0
gptid/b96acf47-ea9a-11e6-b10c-001d091e503b.eli ONLINE 0 0 0

errors: No known data errors

pool: WD-RAIDZ2
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0 days 12:40:25 with 0 errors on Sun Feb 9 12:09:06 2020
config:

NAME STATE READ WRITE CKSUM
WD-RAIDZ2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/73276c84-d0e5-11e8-83ad-7085c28f99a9.eli ONLINE 0 0 0
gptid/74360b24-d0e5-11e8-83ad-7085c28f99a9.eli ONLINE 0 0 0
gptid/754f12ba-d0e5-11e8-83ad-7085c28f99a9.eli ONLINE 0 0 0
gptid/765961f4-d0e5-11e8-83ad-7085c28f99a9.eli ONLINE 0 0 0
gptid/777dda25-d0e5-11e8-83ad-7085c28f99a9.eli ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da7p2 ONLINE 0 0 0

errors: No known data errors


Result of "dmesg" during import of the pool:

GEOM_ELI: Device gptid/777dda25-d0e5-11e8-83ad-7085c28f99a9.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device gptid/754f12ba-d0e5-11e8-83ad-7085c28f99a9.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device gptid/73276c84-d0e5-11e8-83ad-7085c28f99a9.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device gptid/765961f4-d0e5-11e8-83ad-7085c28f99a9.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device gptid/74360b24-d0e5-11e8-83ad-7085c28f99a9.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_MIRROR: Cannot open consumer da4p1 (error=1).
GEOM_MIRROR: Cannot open consumer da3p1 (error=1).
GEOM_MIRROR: Device swap0 destroyed.
GEOM_MIRROR: Cannot open consumer da4p1 (error=1).
GEOM_MIRROR: Cannot open consumer da3p1 (error=1).
GEOM_MIRROR: Device swap1 destroyed.
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_MIRROR: Cannot open consumer da2p1 (error=1).
GEOM_MIRROR: Cannot open consumer da1p1 (error=1).
GEOM_MIRROR: Device swap1 destroyed.
GEOM_MIRROR: Cannot open consumer da2p1 (error=1).
GEOM_MIRROR: Cannot open consumer da1p1 (error=1).
GEOM_MIRROR: Device swap2 destroyed.
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware


Result of "zpool import" didn't return anything.

Result of "sqlite3 /data/feenas-v1.db 'select * from storage_volume;'":

root@freenas-E3[/]# sqlite3 /data/feenas-v1.db 'select * from storage_volume;'
Error: no such table: storage_volume


See the "zfs_info_scrub.png", there has been a srcub of the pool, but I don't know if I trust the result as the pool isn't scrubbing and duration was short, yet the pool doesn't exist in the GUI.
 
Joined
Oct 18, 2018
Messages
969
Result of "sqlite3 /data/feenas-v1.db 'select * from storage_volume;'":
I must've typoed? Also, the 11.3-RELEASE system I'm using was upgrade from 11.2-U7, not sure if that makes any difference. Below is an exact copy-paste from my machine for the command I suggested above.
Code:
baret# sqlite3 /data/freenas-v1.db 'select * from storage_volume;'
35|pool1|13307071281132267760|2|b0fb990f-e4bd-45de-b969-0e14321f5b55
36|pool2|15703373112174060534|2|28g32621-4d58-4352-82fa-1e08702d7b02


Strange that your GUI is showing UNKNOWN on the pools screen. What were the exact steps you took that got you in this situation? I wonder if I can repro it on my system. Sorry if you've said above, I'm having a hard time following every single step. :)
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
Thanks guys... a lot of new information here

I have a second/identical Dell r720xd that I racked up and am going to install a fresh install of Freenas 11.3 to do some testing. First with test with some random drives and then try to import my 'eightTB.spinners' pool which is a simple 3-disk raid-z1. I will leave my 'tenTB.spinners' pool for later because it is a 12 drives (6-disk raid-z2 + 6-disk raid-z2, striped together).

Everything below is from the original FreeNAS system

@PhiloEpisteme asked if I can see the disks and the answer is yes:

These are the disks for pool 'eightTB.spinners'
.. (one raid-z1 + encrypted with password)
.. (the physical disks also show up in the webGUI)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 17 lun 0 (pass20,da19)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 18 lun 0 (pass21,da20)
<ATA WDC WD80EFZX-68U 0A83> at scbus1 target 19 lun 0 (pass22,da21)

These are the disks for pool 'tenTB.spinners'
.. (two raid-z2 vdevs sriped together + encrypted with password)
.. (the Seagate ST1000... drives in one vdev; WD drives in the other)
.. (the physical disks also show up in the webGUI)
<ATA ST10000NM0086-2A SN05> at scbus0 target 9 lun 0 (pass0,da0)
<ATA ST10000NM0086-2A SN05> at scbus0 target 10 lun 0 (pass1,da1)
<ATA ST10000NM0086-2A SN05> at scbus0 target 11 lun 0 (pass2,da2)
<ATA ST10000NM0086-2A SN05> at scbus0 target 12 lun 0 (pass3,da3)
<ATA ST10000NM0086-2A SN05> at scbus0 target 13 lun 0 (pass4,da4)
<ATA ST10000NM0086-2A SN05> at scbus0 target 14 lun 0 (pass5,da5)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 18 lun 0 (pass6,da6)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 19 lun 0 (pass7,da7)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 20 lun 0 (pass8,da8)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 21 lun 0 (pass9,da9)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 22 lun 0 (pass10,da10)
<ATA WDC WD100EFAX-68 0A83> at scbus0 target 23 lun 0 (pass11,da11)

@PhiloEpisteme said:
.. zpool status will not show pools not know to the system
.. which means locked pool will not be available
.. zpool import will not work if drives are locked

>> All this is good news to me <<
>> I just need to figure out procedure to import an encrypted+password pool on Freenas 11.3 <<

@Apollo said he could see encrypted pools using 'zpool list' but I do not. All I see is my unencrypted 'freenas-boot' pool:

root@freenas[/dev/gptid]# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:27 with 0 errors on Fri Feb 7 03:46:29 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da22p2 ONLINE 0 0 0
da23p2 ONLINE 0 0 0

errors: No known data errors

@PhiloEpisteme said:
.. You can verify your pool yourself by trying to manually unlock the pool using:
$ geli attach -k <key> /dev/gptid/<device>

.. but I have 16 devices in there (below)
.. do you know if this command might be destructive?
.. I think I am going to hold off on this one for a bit
root@freenas[/dev/gptid]# ls -al
total 1
dr-xr-xr-x 2 root wheel 512 Feb 9 00:28 .
dr-xr-xr-x 17 root wheel 512 Feb 9 00:28 ..
crw-r----- 1 root operator 0x125 Feb 9 00:28 0d47ded6-fe94-11e9-9e4a-ecf4bbe54910
crw-r----- 1 root operator 0x117 Feb 9 00:33 d95d7fd7-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x123 Feb 9 00:28 d9ced662-ebc6-11e9-8136-ecf4bbe54910
crw-r----- 1 root operator 0x115 Feb 9 00:33 dbb005b9-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x116 Feb 9 00:28 dbd9edca-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x13a Feb 9 00:33 ddd0f056-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x122 Feb 9 00:28 decf653c-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x119 Feb 9 00:33 df421846-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x13c Feb 9 00:33 df4f0a82-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x11b Feb 9 00:33 df84e542-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x11f Feb 9 00:33 dfa0a157-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x11e Feb 9 00:28 e174a3c4-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x118 Feb 9 00:28 e264a677-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x11a Feb 9 00:28 e272f10b-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x120 Feb 9 00:28 e2c9df91-4aa9-11ea-a097-ecf4bbe54910
crw-r----- 1 root operator 0x11c Feb 9 00:28 e2e09d4b-4aa9-11ea-a097-ecf4bbe54910

@PhiloEpisteme question: Can you clarify and post what you get with both zpool status and zpool import, which pool is the one in question, and whether you expect it to be locked, imported, etc?

root@freenas[/dev/gptid]# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:27 with 0 errors on Fri Feb 7 03:46:29 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da22p2 ONLINE 0 0 0
da23p2 ONLINE 0 0 0

errors: No known data errors

(nothing on zpool import)
root@freenas[/dev/gptid]# zpool import
root@freenas[/dev/gptid]#

Both eightTB.spinners (raid-z1) and tenTB.spinners (2x raid-z2) are encrypted with geli.key file and password.

BTW... the keys are still in there:
root@freenas[/data/geli]# ls -al
total 5
drwxrwxrwx 2 root www 4 Feb 8 11:49 .
drwxr-xr-x 8 www www 13 Feb 9 14:31 ..
-rw-r--r-- 1 root www 64 Jan 29 01:04 0c611a6c-00b0-4aff-a50e-7185382754ad.key
-rw-rw-rw- 1 root www 64 Jan 22 21:34 f172b9d1-f862-4a6c-84fc-686822681c44.key
 
Joined
Oct 18, 2018
Messages
969
Everything below is from the original FreeNAS system
What do you mean by that? I'm interested in seeing the output of those commands on the system and version you're experiencing the issue with.

@PhiloEpisteme said:
.. You can verify your pool yourself by trying to manually unlock the pool using:
$ geli attach -k <key> /dev/gptid/<device>

.. but I have 16 devices in there (below)
.. do you know if this command might be destructive?
.. I think I am going to hold off on this one for a bit
This command is NOT destructive. For encrypted disks they are "locked" using geli. When you import or unlock an encrypted pool it uses the exact same command I listed. You can check each of your devices with that command and then see if you see your pool with zpool import. If you do, I think this is a GUI issue for sure.

@PhiloEpisteme question: Can you clarify and post what you get with both zpool status and zpool import, which pool is the one in question, and whether you expect it to be locked, imported, etc?
What I expect is that zpool status shows every pool listed in Storage->Pools AND the boot pool. I do not expect it to list any unimported pools whether they are locked or unlocked.

I expect zpool import to list unimported pools which are NOT locked. I do not expect to see locked pools on this list. Every pool here should also show up in the import non-encrypted pool drop down list.

@PhiloEpisteme said:
.. zpool status will not show pools not know to the system
.. which means locked pool will not be available
.. zpool import will not work if drives are locked

>> All this is good news to me <<
>> I just need to figure out procedure to import an encrypted+password pool on Freenas 11.3 <<

@Apollo said he could see encrypted pools using 'zpool list' but I do not. All I see is my unencrypted 'freenas-boot' pool:
My general sense is that I may not have fully understood @Apollo's case. Apollo is quite knowledgeable and it seems like there might be a UI bug in the mix here. I do still think one should expect the output I described above from those two commands and if there is a mismatch I would guess it indicates either a bug or someone importing/managing pools from the CLI.

BTW... the keys are still in there:
Code:
root@freenas[/data/geli]# ls -al
. . .
-rw-r--r-- 1 root www 64 Jan 29 01:04 0c611a6c-00b0-4aff-a50e-7185382754ad.key
-rw-rw-rw- 1 root www 64 Jan 22 21:34 f172b9d1-f862-4a6c-84fc-686822681c44.key
My pools all show keys with the following permissions -rw-r--r-- 1 root www . Perhaps unrelated, but interesting nonetheless.

Code:
(nothing on zpool import)
root@freenas[/dev/gptid]# zpool import
root@freenas[/dev/gptid]#
This is what I would expect if your pool is still "locked"

Code:
root@freenas[/dev/gptid]# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:27 with 0 errors on Fri Feb 7 03:46:29 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da22p2 ONLINE 0 0 0
da23p2 ONLINE 0 0 0

errors: No known data errors
This is what I would expect if tenTB.spinners is locked.

If you see the devices for eightTB.spinners, see it with camcontrol devlist but do not see it under the import encrypted pool my guess is this is some sort of bug.

What do you get with sqlite3 /data/freenas-v1.db 'select * from storage_volume;'?
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
I must've typoed? Also, the 11.3-RELEASE system I'm using was upgrade from 11.2-U7, not sure if that makes any difference. Below is an exact copy-paste from my machine for the command I suggested above.
Code:
baret# sqlite3 /data/freenas-v1.db 'select * from storage_volume;'
35|pool1|13307071281132267760|2|b0fb990f-e4bd-45de-b969-0e14321f5b55
36|pool2|15703373112174060534|2|28g32621-4d58-4352-82fa-1e08702d7b02


Strange that your GUI is showing UNKNOWN on the pools screen. What were the exact steps you took that got you in this situation? I wonder if I can repro it on my system. Sorry if you've said above, I'm having a hard time following every single step. :)

This is the result I got:


1|HGST-10TB-M|18361101593448120053|0|
3|SSD-Samsung-256GB|16363495742609563155|0|
8|SG-RAIDZ2-1|7028857175536165607|1|3e791190-7177-4b4b-bc09-5e28e08961a3


The steps I used were as follow:

I tried with frsh install of 11.2-U6, then 11.2-7 and then 11.3 RC2 and 11.3 Release and I wasn't able to get proper import. The system is curently located in another location and I should have access to it sometime this week. I should have a backup of the latest config over there so I might be fine doing another fresh install and restore the config.
However, as it is, I have tried importing the pool for WD-RAIDZ2 without success. The steps I have followed are as follow:

Select the import encrypted disk from the GUI. From another instance of the Web interface, I displayed the disk allocation and serial number so I could tel which disk is part of the proper pool.

So I select the disk one by one from the list and then upload the geli.key file. I press next and in the Pool drop-down the pool name would appear. I select it and then proceed with the import. After a while, I get an error message in the GUI.

What I have seen a few times in earlier version of Freenas, such as Coral until now, is that the GUI can fail if volumes defined in the GUI but missing from the system would cause encryption disk/pool import to fail.
Maybe having more than one encrypted pool in the system is also causing the problem.
 

theyost

Dabbler
Joined
Feb 24, 2019
Messages
30
What do you mean by that? I'm interested in seeing the output of those commands on the system and version you're experiencing the issue with.

I am fortunate to have twin hardware to tinker with so I thought I would start fresh and maybe practice there. Both machines are Dell r720xd with JBOD controllers. The old/broken FreeNAS box is sleeping right now.

What is crazy is I can't even create an encrypted pool on a fresh FreeNAS (11.3) install (image downloaded today). I first tried UEFI (set both in hardware and software). The second go-around is with Legacy BIOS (set both in hardware & software). Same results... I a bunch of screenshots just to prove I am not crazy:

Log in first time to fresh FreeNAS 11.3 install and then...
storage->pools
2020.02.09.at.18.41.53.ScreenShot.from.RYZEN-2700X.png


Click "Add" + "Create new pool"
2020.02.09.at.18.41.59.ScreenShot.from.RYZEN-2700X.png


Name: fourTBs
Encryption: <yes>
Warning: <confirm> and <I understand>
2020.02.09.at.18.42.20.ScreenShot.from.RYZEN-2700X.png


Move three disks over
Default is raid-z. I am okay with that
Click "Create"
2020.02.09.at.18.42.38.ScreenShot.from.RYZEN-2700X.png


<confirm> warning + <create pool>
2020.02.09.at.18.42.45.ScreenShot.from.RYZEN-2700X.png


& I get an error.
2020.02.09.at.18.43.03.ScreenShot.from.RYZEN-2700X.png


Error: Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/tastypie/resources.py", line 219, in wrapper
response = callback(request, *args, **kwargs)

File "./freenasUI/api/resources.py", line 1421, in dispatch_list
request, **kwargs

File "/usr/local/lib/python3.7/site-packages/tastypie/resources.py", line 450, in dispatch_list
return self.dispatch('list', request, **kwargs)

File "./freenasUI/api/utils.py", line 252, in dispatch
request_type, request, *args, **kwargs

File "/usr/local/lib/python3.7/site-packages/tastypie/resources.py", line 482, in dispatch
response = method(request, **kwargs)

File "/usr/local/lib/python3.7/site-packages/tastypie/resources.py", line 1384, in post_list
updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))

File "/usr/local/lib/python3.7/site-packages/tastypie/resources.py", line 2175, in obj_create
return self.save(bundle)

File "./freenasUI/api/utils.py", line 493, in save
form.save()

File "./freenasUI/storage/forms.py", line 282, in save
return False

File "./freenasUI/storage/forms.py", line 279, in save
}, job=True)

File "/usr/local/lib/python3.7/site-packages/middlewared/client/client.py", line 513, in call
return jobobj.result()

File "/usr/local/lib/python3.7/site-packages/middlewared/client/client.py", line 276, in result
raise ClientException(job['error'], trace={'formatted': job['exception']})

middlewared.client.client.ClientException: [Errno 17] File exists: '/data/geli'

I am without words. When it rains it pours.
Could it be a driver issue?
PhiloEpisteme can you... can anybody re-create?
Could this be related to my original problem of not being able to unlock encrypted+password pools after reboot?

-Dave
 

Attachments

  • 2020.02.09.at.18.42.58.ScreenShot.from.RYZEN-2700X.png
    2020.02.09.at.18.42.58.ScreenShot.from.RYZEN-2700X.png
    590.2 KB · Views: 234
Joined
Oct 18, 2018
Messages
969
So I select the disk one by one from the list and then upload the geli.key file. I press next and in the Pool drop-down the pool name would appear. I select it and then proceed with the import. After a while, I get an error message in the GUI.
This to me says your disks were successfully unlocked with the key you provided. Am I correct that the issue you're experiencing is with WD-WAIDZ2 and that above you were seeing that pool in the GUI?

Just for giggles to make sure I understand the output well enough what do you get if you run the following commands within sqlite3 to see the headers.
Code:
# sqlite3 /data/freenas-db.v1
sqlite> .mode column
sqlite> .header on
sqlite> select * from storage_volume;


Anyway, I'm not sure what help I can provide for your case anymore. If it were my pool I'd try to check that prior to import that my disks/volume are not found in the database anywhere, specifically in storage_volume and select * from storage_encrypteddisk;. Then, I'd try to import it myself via the CLI by unlocking first and then importing to see if I can import. This would tell me that zfs has no issue with the pool and then if that works I would export the pool and file a bug. If you decide to try that debug route I'd be interested to see the output.

Code:
pool: WD-RAIDZ2
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0 days 12:40:25 with 0 errors on Sun Feb 9 12:09:06 2020
config:

What version of FreeNAS was this pool made in? Did I miss that from above?

I've been trying to repro issues on 11.3-RELEASE but can't :/
 
Joined
Oct 18, 2018
Messages
969
I am without words. When it rains it pours.
Could it be a driver issue?
PhiloEpisteme can you... can anybody re-create?
Could this be related to my original problem of not being able to unlock encrypted+password pools after reboot?
I am curious if your issue has to do with the following line in your config.

middlewared.client.client.ClientException: [Errno 17] File exists: '/data/geli'
What if you do the following . . .

Go ahead and check the output of sqlite3 /data/freenas-v1.db 'select * from storage_volume;'. I expect that you'll see only one for tenTB.spinners. Compare that against the files you see in /data/geli. Whichever of the files in [/icode]/data/geli[/icode] that you do NOT see in the output of the sqlite3 command go ahead and rename them as follows, mv /data/geli/<key>.key /data/geli/<key>.key.bak. This is not destructive, you can always move it back. Try that and then try creating the pool.
edit: corrected typo
 
Last edited:
Top