Jail networking differences between 11.2-U5 and U6/U7?

[fracai]

I'm currently running 11.2-U5 with a number of iocage jails and two legacy jails. All have been running fine, though sorely in need of some updates. A while back I upgraded to U6 and immediately rolled back as all my jails were lacking network access. Since then I haven't had the availability to investigate and figure out what went wrong or if it's a configuration change that would be needed. I seem to remember some discussion about additional devfs privileges that may or may not be required.

I'm planning to update to U7 this weekend and wanted to survey the scene to see what I'm in for.

So, since U5 have there been any changes that would explain the network drop out that I saw? Or is that more likely up to a bad upgrade or a temporary fluke that would have been resolved by restarting the jails or the host system?
Does anyone else recall the devfs changes I think I remember? Were those related to networking? Or am I mis-remembering a discussion about something like access to other hardware components?

Thanks

 

[dlavigne]

Any changes would be in the ticket section of the Release Notes: https://www.ixsystems.com/blog/knowledgebase_category/freenas-release-notes/. Since you saw the behavior on U6, start with those notes (there are some plugins changes there).

 

[fracai]

Thanks, I took a look and nothing jumps out or looks relevant after looking deeper at the tickets. Short of someone having other knowledge about a quirky side effect change or something I'm going to assume my issue was temporary and plow ahead. I'd assume an intended change or a bug causing what I experienced would have generated more attention.
Thanks again.

 

[dlavigne]

Let us know how it goes. If it turns out to be a config issue, someone may know of a workaround.

 

[fracai]

I just upgraded and I've got the same issue. The symptom is that none of the jails startup. Or rather it looks like the first tries to start and gets stuck.
Running iocage list gives a permission-based stack trace. If I run it with sudo I get output so I guess iocage needs root now. That's fine. The one that tried to start has "DHCP(Network Issue)" in the IP4 column.

Any tips?

 

[fracai]

I tried to start one manually and got this output:

Code:

* Starting genericajailname
  + Started OK
  + Configuring VNET OK
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/start.py", line 54, in cli
    ioc.IOCage(jail=jail, rc=rc).start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1641, in start
    callback=self.callback
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 67, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 453, in __start_jail__
    addr = f"{out.splitlines()[2].split()[1].decode()}"
IndexError: list index out of range

 

[fracai]

When running iocage list without root I just noticed that there's a message before any permissions error.
"You need to be root to convert the configurations to the new format!"
This happens every time. Even after running sudo iocage list. Is there a format conversion that was missed during the upgrade? Is this something I could run myself? The full stack trace from iocage list is:

Code:

You need to be root to convert the configurations to the new format!
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1901, in json_check_default_config
    default_props, default=True)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 899, in json_check_config
    silent=self.silent)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 90, in logit
    _callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 73, in callback
    raise SystemExit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/list.py", line 81, in cli
    dataset_type, header, _long, _sort, plugin=plugins, quick=quick)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1169, in list
    quick=quick
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 64, in list_datasets
    iocage_lib.ioc_json.IOCJson().json_check_default_config()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1924, in json_check_default_config
    self.json_write(default_props, default_json_location)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 447, in json_write
    'w') as out:
  File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 512, in open_atomic
    with tempfile(dir=os.path.dirname(os.path.abspath(filepath))) as tmppath:
  File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 475, in tempfile
    tf = tmp.NamedTemporaryFile(delete=False, suffix=suffix, dir=dir)
  File "/usr/local/lib/python3.6/tempfile.py", line 551, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/local/lib/python3.6/tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/mnt/iocage/tmp1aavajzx'

 

[fracai]

So I noticed that I have two bridge interfaces: 0 and 1. My legacy jails are connected to 0 and the iocage jails are configured for 1. I seem to remember doing something with this a while ago where the bridge interfaces kept switching every boot. For now I've changed the interfaces setting on every iocage jail from vnet0:bridge1 to vnet0:bridge0 and they're starting up.

 

[fracai]

All the release upgrades went fine. I'll figure out the bridge stuff the next time I reboot.

For now my only question is about the root requirement for running iocage list.
- Is that a new requirement?
- Is the reference to configuration conversion something that needs to be done?

 

[dlavigne]

All the release upgrades went fine. I'll figure out the bridge stuff the next time I reboot.

For now my only question is about the root requirement for running iocage list.
- Is that a new requirement?
- Is the reference to configuration conversion something that needs to be done?

I vaguely remember seeing a commit go by to that effect, but couldn't find the related ticket from a quick Jira search... If you happen to have better luck than me, post the ticket number for the curious.