Partial error when importing an encrypted volume

[booya]

Hi everyone!

I've been experiencing some errors when importing an encrypted volume on FreeNAS 9.10.2 U2, U3, U5 and now 11.0-RELEASE.

Second level dirs and below does not mount. I believe that this error stems from it being a read-only volume (used as an off-site replication target).

So I wonder, is this by "design" or is it something that can be fixed? Replication does seemingly still work. Maybe I'm going out on a limb here but could it be possible to fix this by changing the read-only attribute?

Code:

Request Method:	POST
Request URL:	https://192.168.7.10/storage/auto-import/
Software Version:	FreeNAS-11.0-RELEASE (a2dc21583)
Exception Type:	MiddlewareError
Exception Value:	

[MiddlewareError: b'The volume "slowraidz2" failed to import, for futher details check pool status']

Exception Location:	./freenasUI/middleware/notifier.py in volume_import, line 3116
Server time:	Fri, 16 Jun 2017 16:06:11 +0000
Traceback
Environment:

Software Version: FreeNAS-11.0-RELEASE (a2dc21583)
Request Method: POST
Request URL: https://192.168.7.10/storage/auto-import/

Code:

Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  39.  response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.  response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.  response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.  return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.  return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in view
  68.  return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in dispatch
  237.  response = super(WizardView, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in dispatch
  88.  return handler(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in post
  300.  return self.render_done(form, **kwargs)
File "/usr/local/lib/python3.6/site-packages/formtools/wizard/views.py" in render_done
  357.  **kwargs)
File "./freenasUI/storage/forms.py" in done
  877.  self.volume = notifier().volume_import(vol['label'], vol['id'], key, passphrase, enc_disks)
File "./freenasUI/middleware/notifier.py" in volume_import
  3116.  'for futher details check pool status') % volume_name)

Exception Type: MiddlewareError at /storage/auto-import/
Exception Value: [MiddlewareError: b'The volume "slowraidz2" failed to import, for futher details check pool status']

Code:

Request information
GET

No GET data
POST
Variable	Value
auto_import_wizard-current_step	'2'
2-__all__	''
2-volume_id	'slowraidz2|7473525507379358860'
__form_id	'dijit_form_Form_2'
FILES

No FILES data
COOKIES
Variable	Value
csrftoken	'xyz'
fntreeSaveStateCookie	'root%2Croot%2F1%2F5'
sessionid	'xyz'
META
Variable	Value

Code:

Jun 16 16:06:11 freenas2 uwsgi: [middleware.notifier:2975] Importing slowraidz2
[7473525507379358860] failed with: cannot mount '/mnt/slowraidz2/backup/dc': fai
led to create mountpoint 
cannot mount '/mnt/slowraidz2/backup/old': failed to create mountpoint 
cannot mount '/mnt/slowraidz2/backup/old/dc': failed to create mountpoint 
cannot mount '/mnt/slowraidz2/backup/time': failed to create mountpoint
cannot mount '/mnt/slowraidz2/users/user0': failed to create mountpoint 
cannot mount '/mnt/slowraidz2/use 
Jun 16 16:06:11 freenas2 uwsgi: rs/user1': failed to create mountpoint 
cannot mount '/mnt/slowraidz2/users/user2': failed to create mountpoint

OT: Has it always been the default to automatically send crash reports?

Code:

Jun 16 16:06:11 freenas2 uwsgi: [raven.base.Client:214] Configuring Raven for host: <raven.conf.remote.RemoteConfig object at 0x81ad07588>
Jun 16 16:06:11 freenas2 uwsgi: [middlewared.logger.CrashReporting:84] Sending a crash report...
Jun 16 16:06:11 freenas2 uwsgi: [raven.base.Client:635] Sending message of length 5781 to https://sentry.ixsystems.com/api/2/store/

It shall be noted that this volume (the disks) was moved off-site (post replication) to another FreeNAS system. The original system was shut down before the removal of the disks but no other actions were performed.

 

[dlavigne]

Were you able to resolve this?

 

[perspectives fran]

I face exactly the same problem and can't solve it. Were you able to solve it ?

 

[booya]

Hi again,

I'm sorry for the late answer but work came in the way! :)

As suspected the read-only attribute prevented second level dirs and below from mounting, so as a quick fix all I did was: "zfs set readonly=off dataset/folder1/folder2" to all second level dirs and below:

Code:

root@freenas2:~ # zfs set readonly=off slowraidz2/backup/time
root@freenas2:~ # zfs set readonly=off slowraidz2/backup/old
root@freenas2:~ # zfs set readonly=off slowraidz2/backup/old/dc

Shouldn't importing an encrypted volume work out of the box? Is this some kind of regression?

I ran into some issues after the quick fix:
ZFS boot error=63 "Unable to create ZVOL" after upgrade 9.10->11.0. Cosmetic?
Replication causing datasets to act unmounted
Thankfully updating to 11.0-U1 seem to have solved these.

Now I only have this issue:
collectd user object callback failed" log spam at least once a day

Lastly, seeing this thread and this and the posts by depasseg:

Code:

"I think it makes sense for replication targets to be mounted read only. This avoids the situation where 2 folders get out of sync, and the changes on the replication target get overwritten.

There were some issues (and I believe bugs open) where the fact that replication target being mounted read-write caused problems.

Since ZFS operates at a level lower, the fact that it's mounted as read-only doesn't affect the ZFS replication tasks."

"
There isn't a way within the FreeNAS UI.  In fact there were issues having the replication target read-writable, which is, I believe, why they changed to to be read-only.  You can try using the CLI to unmount and mount re-write, but FreeNAS might change it back (possibly after the next replication or reboot).  Please be aware, that this isn't tested, nor advised and may require you to destroy your replication target and start replication from scratch.
https://docs.oracle.com/cd/E23824_01/html/821-1448/gaynd.html
"

And this thread and the post by cyberjock:

Code:

...
"Now, the replicator FreeNAS uses requires the readonly attribute to be set to on. This prevents Samba, NFS, or iSCSi from trying to initiate a write at the same time as replication, but it does nothing for other scenarios. This has also been a change in behavior that has only recently (2-3 months?) become a problem because of the rewrite of the replicator to allow for new scenarios (A->B and A->C at the same time). "
...

I wonder if my quick fix is acceptable or if it will lead to more issues ahead?

 

[booya]

Update for future reference:

I've just updated to 11.0-U2 and got the following error when trying to decrypt the volume using passphrase/geli-key:

Code:

Environment: Software Version: FreeNAS-11.0-U2 (e417d8aa5) Request Method: POST Request URL: https://192.168.7.10/storage/volume/1/unlock/?X-Progress-ID=a1626fef-b80c-482a-8dac-2281e0b8d2d8 Traceback: File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner 39. response = get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response 249. response = self._get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response 178. response = middleware_method(request, callback, callback_args, callback_kwargs) File "./freenasUI/freeadmin/middleware.py" in process_view 162. return login_required(view_func)(request, *view_args, **view_kwargs) File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view 23. return view_func(request, *args, **kwargs) File "./freenasUI/storage/views.py" in volume_unlock 1190. form.done(volume=volume) File "./freenasUI/storage/forms.py" in done 2616. raise MiddlewareError(msg) Exception Type: MiddlewareError at /storage/volume/1/unlock/ Exception Value: [MiddlewareError: b'Volume could not be imported: 4 devices failed to decrypt']

This was solved by detaching and reimporting the volume.

Using geli-key on two separate volumes on another FreeNAS 11.0-U2 system resulted in a similar error, fortunately using the passphrase still worked.