Plugins Fail to Create after moving Jails to different pool

[Scharbag]

As the post states, I am having issues creating new plugins after I moved my jail install location to a different pool. I followed the post here. My Jail Root is configured as: /mnt/fasttank/Jails

I am running on the latest 9.3 patches (Jan 9, 2015).

Everything seemed to work fine as my jails were relocated and they all launch without issue. I deleted the old jails off of the previously used pool to prove this. When I create a new jail, everything works fine as shown below:

Code:

Jan 11 11:32:49 freenas warden: Building new Jail... Please wait...
Jan 11 11:32:49 freenas warden: zfs clone fasttank/Jails/.warden-template-standard@clean fasttank/Jails/test
Jan 11 11:32:50 freenas warden: Success!
Jan 11 11:32:50 freenas warden: Jail created at /mnt/fasttank/Jails/test

The problem seems to occur when I try to add a new plugin (MediaBrowser, Subsonic etc.). I get the following error:

Code:

Jan 11 09:40:01 freenas warden: Building new Jail... Please wait...
Jan 11 09:40:01 freenas warden: zfs clone fasttank/mnt/fasttank/Jails/.warden-template-pluginjail@clean fasttank/Jails/mediabrowser_1
Jan 11 09:40:01 freenas warden: ERROR: Failed creating clean ZFS base clone
Jan 11 09:40:01 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Failed to install plugin: cannot open 'bigtank
fasttank': invalid dataset name
cannot open 'fasttank/mnt/fasttank/Jails/.warden-template-pluginjail@clean': dataset does not exist
ERROR: Failed creating clean ZFS base clone
]

My old pool where the jails were located was called bigtank. My new pool where the jails are now located is called fasttank. I do have all of the @clean snapshots needed I think:

Code:

fasttank/Jails/.warden-template-VirtualBox-4.3.12_DGS@clean
fasttank/Jails/.warden-template-pluginjail@clean
fasttank/Jails/.warden-template-pluginjail-9.2-RELEASE-x64@clean
fasttank/Jails/.warden-template-standard@clean

The old 9.2 version is for an old jail I still have (just waiting until I prove the new 9.3 jail works for my FlexGet setup).

It looks to me that when it is trying to clone the snapshot, the path is messed up. For some reason, there is a leading fasttank/ before the proper mnt/fasttank/<jailname> call. It may also be an issue that the plugin add procedure still references the old bigtank location for some reason.

I am unsure of how to proceed further with this issue. Any help would be greatly appreciated. I am sure that there is just some reference set somewhere but I am unsure of the location or how to fix it.

And, I cannot recall if I moved the jails before I installed Transmission 2.84 or after. That is the only plugin I currently have running so if it would help to delete that plugin, and remove the plugin jail template, I could easily do that. Might try that ASAP.

If there is some missing information that is needed, please let me know and I will post it right away.

UPDATE: So, I deleted the transmission plugin. When I tried to re-install the plugin, I got the same error as above. To try to force FreeNAS to re-download the plugin jail, I deleted the plugin jail template dataset and directory. Then when I try to add the plugin, I get the following:

Code:

Jan 11 12:31:41 freenas warden: Building new Jail... Please wait...
Jan 11 12:31:41 freenas warden: zfs clone fasttank/Jails/.warden-template-pluginjail@clean fasttank/Jails/transmission_1
Jan 11 12:31:41 freenas warden: ERROR: Failed creating clean ZFS base clone
Jan 11 12:31:41 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Failed to install plugin: cannot open 'fasttank/Jails/.warden-template-pluginjail@clean': dataset does not exist
ERROR: Failed creating clean ZFS base clone

It seems to be really messed up as it also added the following dataset:

Code:

/mnt/fasttank/mnt/fasttank/Jails# ls -l
total 2
drwxr-xr-x  3 root  wheel  3 Jan 11 12:19 ./
drwxr-xr-x  3 root  wheel  3 Jan 11 12:19 ../
drwxr-xr-x  2 root  wheel  2 Jan 11 12:19 .warden-template-pluginjail/

I think I broke it... Any help would be fantastic :)

Cheers,

 

[Scharbag]

I tried to save my debug too but I got this:

Code:

Request Method:    GET
Request URL:    https://10.0.1.210/system/debug/
Software Version:    FreeNAS-9.3-STABLE-201501090144
Exception Type:    IOError
Exception Value:   
[Errno 2] No such file or directory: '/var/db/system/ixdiagnose/ixdiagnose.tgz'
Exception Location:    /usr/local/www/freenasUI/../freenasUI/system/views.py in debug, line 1027
Server time:    Sun, 11 Jan 2015 12:55:04 -0600
Traceback


Request information

GET

No GET data

POST

No POST data

FILES

No FILES data

COOKIES

Variable    Value
sessionid    'wsbr28bvzggh8jx8eej1zeuslxdloiys'
fntreeSaveStateCookie    'root%2Croot%2F17%2F27%2Croot%2F75%2F216%2Croot%2F293%2Croot%2F293%2F317%2Croot%2F293%2F317%2F319%2Croot%2F297'
csrftoken    'KSCG5znFbPk7oq9Y9YuGOKPUqcvphy4m'
META

Variable    Value
wsgi.run_once    False
SERVER_NAME    'localhost'
REMOTE_PORT    '63338'
DOCUMENT_ROOT    '/usr/local/etc/nginx/html'
HTTP_REFERER    'https://10.0.1.210/'
HTTP_COOKIE    'csrftoken=KSCG5znFbPk7oq9Y9YuGOKPUqcvphy4m; fntreeSaveStateCookie=root%2Croot%2F17%2F27%2Croot%2F75%2F216%2Croot%2F293%2Croot%2F293%2F317%2Croot%2F293%2F317%2F319%2Croot%2F297; sessionid=wsbr28bvzggh8jx8eej1zeuslxdloiys'
DOCUMENT_URI    '/system/debug/'
CONTENT_TYPE    ''
HTTP_ACCEPT_LANGUAGE    'en-us'
wsgi.url_scheme    'https'
HTTPS    'on'
REQUEST_METHOD    'GET'
SERVER_PROTOCOL    'HTTP/1.1'
PATH_INFO    u'/system/debug/'
HTTP_HOST    '10.0.1.210'
HTTP_ACCEPT_ENCODING    'gzip, deflate'
wsgi.errors    <flup.server.fcgi_base.TeeOutputStream object at 0x8127b1490>
GATEWAY_INTERFACE    'CGI/1.1'
CSRF_COOKIE    u'KSCG5znFbPk7oq9Y9YuGOKPUqcvphy4m'
SERVER_SOFTWARE    'nginx/1.6.2'
wsgi.multiprocess    False
SCRIPT_NAME    u''
wsgi.version    (1, 0)
HTTP_ACCEPT    'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
wsgi.input    <flup.server.fcgi_base.InputStream object at 0x81622bdd0>
QUERY_STRING    ''
REDIRECT_STATUS    '200'
wsgi.multithread    True
REQUEST_URI    '/system/debug/'
HTTP_USER_AGENT    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5'
REMOTE_ADDR    '10.0.1.142'
SERVER_PORT    '443'
CONTENT_LENGTH    ''
SERVER_ADDR    '10.0.1.210'

 

[Simon Mackenzie]

What you have described makes sense. I had just setup .system on anther volume and changed the JailRoot to this new volume also. Had previous jails on the old volume but had discarded them for a clean plugin install on the new volume. My first plugin install took a very long time but succeeded. My attempt to install a second plugin on the new volume failed. Prior to moving JailRoot it appeared everything was working fine.

 

[Scharbag]

Any idea how to fix this?

 

[Simon Mackenzie]

I don't know the details but it appears to be related to the way plugins and jails are created using snapshots and clones. In my case I don't think this should have been a problem but I'm very new to FreeNAS and its inner workings.

 

[Scharbag]

Well, given that my config has been constantly upgraded from V8.x, I decided to start fresh and set the thing up from scratch again.

Everything seems to work now, and as a bonus, I fixed some permission issues I created for myself when I set it up the first time years ago!!

Now, I think I just need to leave well enough alone FFS!!! :P

Cheers,

 

[Simon Mackenzie]

I'll be doing the same in a day or so.

 

[doctor15]

Does anyone have a solution to this? I'm suffering from the same issue. Surely there must be a way to move the Jails without starting fresh?

 

[CyrilP]

Not 100% sure about that but if you are moving from one pool to another you must transfert everything using zfs send and restore the mount point.

1. zfs snapshot -r main_pool/jails@relocate
   zfs send -R main_pool/jails@relocate | zfs receive -v ssd_pool/jails
   zfs get -rH -o name -s received mountpoint ssd_pool/jails | xargs -I {} sh -c "zfs set mountpoint=/{} {}; zfs mount {};"[/PANEL]
2. Change the Jail Root to /mnt/ssd_pool/jails (Jails->Configuration)
3. Start jails/plugins
4. Check that everything works and destroy the original jails dataset (main_pool/jails)

https://forums.freenas.org/index.php?threads/relocate-jails-to-ssd-helping-hdd-sleep.16955/page-2

Edit: I just figure out that you already do that :p

 

[CyrilP]

You should have try to remove templates using warden. Check if something still rely on the old snapshots with

Code:

zfs list -o origin,name,used,avail,refer,mountpoint

 

[doctor15]

Awesome, this is exactly what I needed!

No, I naievely copied everything using rsync. The Jails run fine though.