Storage Unlock Errors and lost jails after latest update

[bornstellare]

I got a alert today about a new update available. I upgraded my FreeNAS and it is now on FreeNAS-9.10-STABLE-201604181743 (74ef270).

However after the reboot, I got the following error when trying to unlock my zpools.

Environment: Software Version: FreeNAS-9.10-STABLE-201604181743 (74ef270) Request Method: POST Request URL: http://172.16.0.2/storage/volume/3/unlock/?X-Progress-ID=41885e3f-5888-490e-826e-b53a4ba69301 Traceback: File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response 105. response = middleware_method(request, callback, callback_args, callback_kwargs) File "/usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py" in process_view 156. return login_required(view_func)(request, *view_args, **view_kwargs) File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view 22. return view_func(request, *args, **kwargs) File "/usr/local/www/freenasUI/../freenasUI/storage/views.py" in volume_unlock 1128. form.done(volume=volume) File "/usr/local/www/freenasUI/../freenasUI/storage/forms.py" in done 2546. notifier().sync_encrypted(volume=volume) File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in sync_encrypted 4756. ed.encrypted_disk and ed.encrypted_disk.id != disk.id Exception Type: AttributeError at /storage/volume/3/unlock/ Exception Value: 'Disk' object has no attribute 'id'

My system is setup in such a way where my Jails are on one volume (system) and my data/backups on another volume (tank). The system itself has been running flawlessly for two years now.

freenas# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 14.9G 10.5G 4.39G - - 70% 1.00x ONLINE -
system 696G 111G 585G - 15% 15% 1.00x ONLINE /mnt
tank 10.9T 6.46T 4.41T - 19% 59% 1.00x ONLINE /mnt


Unlocking both volumes resulted in the same error. My tank zpool seems to be intact (I am still going through it via CLI since my shares are also not available). I can't see any of my jails though. The system volume also stored RRD and none of the reporting data is displaying.

I am currently looking at ways to move the data in tank somewhere. I have a backup of the important data with a potential loss of 2-3 days, so its no big deal.

Is there any way I can recover the system volume and find out what happened?

[root@freenas ~]# warden list
Error: No jails found!

freenas# ls /mnt/system/
.system Jails

The Jails configs seem to be there on the drive.

freenas# ls -al /mnt/system/Jails/
total 214
drwxr-xr-x 28 root wheel 28 Apr 20 21:35 .
drwxrwxr-x 4 root wheel 4 Apr 20 21:35 ..
drwxr-xr-x 2 root wheel 17 Jun 9 2015 .VM.meta
drwxr-xr-x 2 root wheel 17 Jun 9 2015 .VirtualBox.meta
[..]
drwxr-xr-x 3 www www 3 Feb 21 2015 .plugins
drwxr-xr-x 5 root wheel 5 Jun 22 2015 .warden-files-cache
drwxr-xr-x 17 root wheel 23 Apr 18 2015 .warden-template-VirtualBox-4.3.12
drwxr-xr-x 18 root wheel 22 Apr 17 21:37 .warden-template-pluginjail
drwxr-xr-x 18 root wheel 22 Feb 21 2015 .warden-template-pluginjail--x64
drwxr-xr-x 18 root wheel 22 Jun 22 2015 .warden-template-pluginjail--x64-20150623131951
drwxr-xr-x 18 root wheel 22 Sep 12 2015 .warden-template-pluginjail-9.3-x64
drwxr-xr-x 17 root wheel 21 Apr 5 2015 .warden-template-standard--x64
drwxr-xr-x 17 root wheel 23 Apr 18 2015 VM
drwxr-xr-x 17 root wheel 22 Jun 28 2015 VirtualBox
[..]

 

[dlavigne]

Please create a bug report at bugs.freenas.org and post the issue number here. Include a debug in the report (created using System -> Advanced -> Save Debug).

 

[SweetAndLow]

Roll back the boot environment, that will probably fix everything.

 

[bornstellare]

Thanks for the recommendation. I have filed https://bugs.freenas.org/issues/14887

I will try changing the boot-env tonight and see if that helps.

Cheers.

 

[onur]

I had the same problem. After upgrading to 9.10 latest stable, and unlocking the storage I am having same error, and my jail is gone (owncloud).

Onur

 

[trinxie]

I got the same problem today :-(
A workaround is to unlock once more and then restart the CIFS service.
Should i rollback or wait for an update?
/Jörgen

 

[trinxie]

Sorry forgot to mention version used: FreeNAS-9.10-STABLE-201604181743 (74ef270)
Forgot to check... : It seems like a patch is available Bug #14789
I will check the bug reports better next time...

 

[bornstellare]

OK, so reverting back to an earlier boot environment fixed the issue for me. I got my precious jails back.

I reverted to FreeNAS-9.10-STABLE-201604111739 (896cc83)

You can do this from System > Boot. Select one of the earlier boot options. Mine is "9.10-STABLE-201604111739". Click on "Activate" and give it around 5 minutes to apply. Once it says "On Reboot", you should be able to reboot your FreeNAS Box.

Thanks all for your help. :) Much appreciated.