SOLVED Jail mountpoints removal kills VNET

proto

Patron
Joined
Sep 28, 2015
Messages
269
Umm...
That's strange:

I've installed CLAMAV plugin, configured static VNET ip, added two RO mountpoints from storage.
Jail is starting and working until I do these steps:
1. stop clamav jail/plugin
2. delete one mountpoint
3. start jail/plugin

At step 2: deleting one mountpoint deletes *all* configured mountpoints

At step 3: jail cannot start again due to VNET failure.

iocage.log:

Code:
2019/03/04 11:10:15 (INFO) * Starting clamav
2019/03/04 11:10:15 (INFO)   + Started OK
2019/03/04 11:10:15 (INFO)   + Configuring VNET OK
2019/03/04 11:10:35 (INFO)   + Starting services OK
2019/03/04 11:15:03 (INFO) * Stopping clamav
2019/03/04 11:15:03 (INFO)   + Running prestop OK
2019/03/04 11:15:04 (INFO)   + Stopping services OK
2019/03/04 11:15:04 (INFO)   + Tearing down VNET OK
2019/03/04 11:15:04 (INFO)   + Removing devfs_ruleset: 10 OK
2019/03/04 11:15:04 (INFO)   + Removing jail process OK
2019/03/04 11:15:04 (INFO)   + Running poststop OK
2019/03/04 11:19:18 (ERROR)
Stopped clamav due to VNET failure
2019/03/04 11:19:35 (INFO) * Starting clamav
2019/03/04 11:19:35 (INFO)   + Started OK
2019/03/04 11:19:35 (ERROR)   + Configuring VNET FAILED
2019/03/04 11:19:35 (ERROR)   jexec: execvp: ifconfig: No such file or directory
2019/03/04 11:19:35 (ERROR)   jexec: execvp: ifconfig: No such file or directory
2019/03/04 11:19:35 (ERROR)
Stopped clamav due to VNET failure


Below the python exception found in middlewared.log / NUW UI:

[2019/03/04 10:33:05] (WARNING) application.call_method():199 - Exception while calling jail.start(*['clamav'])
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 128, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 88, in _run
return await self._call(f'{service_name}.{method}', serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 581, in start
iocage.start()
File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1663, in start
callback=self.callback
File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 66, in __init__
self.__start_jail__()
File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 491, in __start_jail__
_callback=self.callback)
File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 90, in logit
_callback(content, exception)
File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 64, in callback
raise callback_exception(message)
RuntimeError:
Stopped clamav due to VNET failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1141, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1078, in _call
return await self._call_worker(serviceobj, name, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1105, in _call_worker
job,
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1036, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1021, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped clamav due to VNET failure
[2019/03/04 10:33:05] (DEBUG) middlewared.logger.CrashReporting.report():103 - Sending a crash report...
[2019/03/04 10:33:05] (DEBUG) raven.base.Client.send_remote():716 - Sending message of length 2898 to https://sentry.ixsystems.com/api/2/store/


update:

ah, /sbin directory in plugins is empty.

Code:
root@bestia[/mnt/Prigioni/iocage/jails/clamav/root]# find . -name ifconfig
./usr/tests/sbin/ifconfig
root@bestia[...rigioni/iocage/jails/clamav/root/sbin]# ls -la
total 9
drwxr-xr-x   2 root  wheel   2 Mar  4 11:05 .
drwxr-xr-x  18 root  wheel  23 Mar  4 11:12 ..
 
Last edited:

tonyxx

Cadet
Joined
Mar 4, 2019
Messages
3
em........



yes I did the same thing in the jail of transmission , and get the this result , the transmission cannot startup because the vet0 failure

I don't think this is the router problem because in the router I have bound the transmission jail mac to the ip address I give it.

what can I do now?

thanks.

Code:
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 128, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 88, in _run
    return await self._call(f'{service_name}.{method}', serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 581, in start
    iocage.start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1663, in start
    callback=self.callback
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 66, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 491, in __start_jail__
    _callback=self.callback)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 90, in logit
    _callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 64, in callback
    raise callback_exception(message)
RuntimeError:
Stopped transmission due to VNET failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1141, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1078, in _call
    return await self._call_worker(serviceobj, name, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1105, in _call_worker
    job,
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1036, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1021, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped transmission due to VNET failure



any advice ??
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
For me it's a bug affecting PLUGINS, not JAILS.
It's not a networking issue but something happening at NEW UI / iocage level...

JAILs work as expected.

I opened this post in the wrong forum section with wrong title. Sorry!
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
Currently, as a workaround, I do not remove mountpoints from the plugin. It is not a "critical" plugin, so I can possibly recreate it if it stops working. It is the only plugin I use.
 

demon

Contributor
Joined
Dec 6, 2014
Messages
117
I ran into this today - I couldn't edit the mountpoint on a jail (the web UI gave me an error), so I deleted and re-added the mountpoint, and apparently that removed most of the files in the jail root. /bin, /sbin and others were devoid of files. Fortunately I was able to remake the jail - the app config was in the mount, so no config data was lost - but that definitely seems like a problem.

Also, I'm unable to edit the jail configuration at all - no matter what changes I make in the web UI, the "Save" button remains grayed out.

I am running FreeNAS 11.2-U2.1.
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
I think it's related to this bug #76864 which should be fixed in U3.
You can also check this one (closed as duplicate #78762)

. /bin, /sbin and others were devoid of files
No custom install on plugins...
 

acp

Explorer
Joined
Mar 25, 2013
Messages
71
I ran into this today - I couldn't edit the mountpoint on a jail (the web UI gave me an error), so I deleted and re-added the mountpoint, and apparently that removed most of the files in the jail root. /bin, /sbin and others were devoid of files. Fortunately I was able to remake the jail - the app config was in the mount, so no config data was lost - but that definitely seems like a problem.

Also, I'm unable to edit the jail configuration at all - no matter what changes I make in the web UI, the "Save" button remains grayed out.

I am running FreeNAS 11.2-U2.1.

If your IP address is set to dhcp, Change the IP address to any valid address. Once done you can save the changes. And the IP address will go away cause, well its DHCP.
 

Andrew Ostrom

Explorer
Joined
Jul 28, 2017
Messages
57
I had this with PLEX today - all my mount points are GONE. Do I just re-create them?

EDIT:

Recreating them didn't help - I still get the VNET error. Any workarounds?
 
Last edited:

proto

Patron
Joined
Sep 28, 2015
Messages
269
Workaround is in #78762 comments and post #5.
I just do not remove mount points once created... it's a middleware issue not networking. But middleware actually seems to break plugins, not jails.
 

grizmo

Cadet
Joined
Mar 23, 2019
Messages
5
I have this same problem here.

I been trying to get permissions working on syncthing. I still can't get it working. So I decided to delete the mount point just because. When I re-run it, it'll give me an error about not able to run /bin/sh. I re-run it, and it says it is running. huh.. strange. Open up the IP address, and it re-routes to freenas login home page...

From that point and on, I no longer able to go to syncthing static IP address . It has been "taken", and simply re-routes to freenas login home page. I create a new static IP address.

I am able to produce this exact same error every single time.

PS!
I did delete my storage pool, and my jail pool, and still get this same error.

I guess the only solution is to nuke it from the orbit.
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
Mountpoint issue is solved in 11.2-U3!
Update at your will!

I set to solved.

Thanks devs!
 

bustersnm

Dabbler
Joined
Apr 2, 2015
Messages
25
I recently added a mount point to my jail, made a mistake and deleted it. Now i get the error below:




Code:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 128, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 88, in _run
    return await self._call(f'{service_name}.{method}', serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 609, in start
    iocage.start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1663, in start
    callback=self.callback
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 66, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 491, in __start_jail__
    _callback=self.callback)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 90, in logit
    _callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 64, in callback
    raise callback_exception(message)
RuntimeError:
Stopped emby-server due to VNET failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1141, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1078, in _call
    return await self._call_worker(serviceobj, name, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1105, in _call_worker
    job,
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1036, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1021, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped emby-server due to VNET failure



emby1.PNG emby2.PNG


Any help would be great.
Thanks.
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
I recently added a mount point to my jail, made a mistake and deleted it. Now i get the error below:

Any help would be great.
Thanks.

Can you try with other plugins? ex. CLAMAV.
Latest update (U3) solved this issue. Anyway, it's a middleware problem, VNET failure is just a consequence, as I understand.
 
Joined
Mar 11, 2019
Messages
5
I'm on U3, installed plex, deleted one of the mountpoints and I got the same error.

Code:
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 128, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 88, in _run
    return await self._call(f'{service_name}.{method}', serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 609, in start
    iocage.start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1663, in start
    callback=self.callback
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 66, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 491, in __start_jail__
    _callback=self.callback)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 90, in logit
    _callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 64, in callback
    raise callback_exception(message)
RuntimeError:
Stopped plexmediaserver-plexpass due to VNET failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1141, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1078, in _call
    return await self._call_worker(serviceobj, name, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1105, in _call_worker
    job,
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1036, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1021, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped plexmediaserver-plexpass due to VNET failure
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
That's strange, because 11.2-U3 really solved that issue (post #1).

I tested again installing PLEX and IRSSI plugins, adding one or two mountpoints, restarting, deleting mountpoints... NO issue.

The only main difference between bustersnm and me is that I performed a clean system install, not an upgrade.
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
If it's a recurring issue, yes. And please submit bug nr. here.

I opened the issue @ release 11.2-U2.1, then I had another problem with a single drive boot so I switched to mirror booting performing a clean install in U3.

If you have some spare time and a VM environment (esx or virtualbox) you could test booting in 11.2-U3 only and installing some plugins.
 
Top