iocage, vnet0, and bridge0

[amlamarra]

This is what got networking to work for me. In the FreeNAS beta GUI, I created a new jail and checked the box for "Vnet". That's it.

 

[dureal99d]

Ive actually found it better and easier to use the cli. in this case this is fail proof for me:

Code:

iocage create -n "yourjailname" -r 11.1-RELEASE ip4_addr="vnet0|192.XXX.X.X/XX" defaultrouter="192.XXX.X.X" vnet="on" allow_raw_sockets="1" boot="on"

this command has been a saver for me and hopefully it works for all

 

[Jailer]

Hello,

i had the same problem but after a few research i have added 2 tunable in the rc.conf :

Code:

Variable : cloned_interfaces ; Value : bridge0
Variable : ifconfig_bridge0 ; Value : addm igb0 up

Thanks @Celena your fix saved me a lot of time and headaches trying to figure out why none of my jails worked after a reboot.

Has anyone filed a bug report on this?

 

[NetSoerfer]

Thanks @Celena your fix saved me a lot of time and headaches trying to figure out why none of my jails worked after a reboot.

Has anyone filed a bug report on this?

Yep, looks like they have: https://redmine.ixsystems.com/issues/33054

 

[Slovak]

Hello,

i had the same problem but after a few research i have added 2 tunable in the rc.conf :

Code:

Variable : cloned_interfaces ; Value : bridge0
Variable : ifconfig_bridge0 ; Value : addm igb0 up

Thank you for posting this, after upgrade to 11.1-U5 I experienced loss of network connectivity for my iocage jails. This solution resolved it.

 

[raidflex]

The below link is very helpful for setting up your networking config with iocage. Good to see this is finally be addressed in 11.1U6.

https://iocage.readthedocs.io/en/latest/networking.html#vimage-vnet

 

[kpfleming]

So best i can tell, bridge1 is the bridge for my functioning Warden jail, with epair1a and lagg0 as members. In that jail, the interface is epair1b.
In the iocage jail, the interface is epair0b. The bridge for this iocage is bridge0, which only has the member vnet0:4. When i try to add the lagg0 to bridge0, i get:

root@freenas:~ # ifconfig bridge0 addm lagg0
ifconfig: BRDGADD lagg0: Device busy

Is it possible to have lagg0 as a member of 2 different bridges?

No, that's not possible. However, your issue has a different cause, which I've run into as I have almost the the same configuration.

The issue is that the FreeNAS link aggregation configuration is *not* handled by setting rc.conf variables; instead the aggregations are created by a script which runs *after* rc.conf is done. As a result, anything you do in rc.conf (system tunables in FreeNAS) which attempts to refer to lagg0 or any other link aggregation fails because it doesn't exist at the point where rc.conf is running. This is a really tricky problem to solve, as it's based on the dependencies between various bits of configuration. It's even trickier for me, as I wanted to create VLAN subinterfaces on top of the aggregation, and then allow jails to use them.

In the end I stopped using bridges, and used a feature of the underlying jail system (the vnet.interface jail property) which allows the jail to steal an interface from the host at startup, and then return it at shutdown. No need for a bridge at all, then, if you have dedicated interfaces (physical or VLAN) for the jails to use.

 

[fd0]

No, that's not possible. However, your issue has a different cause, which I've run into as I have almost the the same configuration.

In the end I stopped using bridges, and used a feature of the underlying jail system (the vnet.interface jail property) which allows the jail to steal an interface from the host at startup, and then return it at shutdown. No need for a bridge at all, then, if you have dedicated interfaces (physical or VLAN) for the jails to use.

Could you share some info on this, how you achieved this?
TIA

 

[kpfleming]

When you create the jail with the iocage command (CLI, not in the GUI), supply "vnet=on" and "vnet_interfaces='<interface_name>'". This won't stop iocage from creating bridge0 and vnet0 (some day I'll send a patch to fix that), but it will *move* the identified interface into the jail when it starts up, and (usually) remove it when the jail is shut down.

Instead of using the ip4_addr/ip6_addr properties, which won't be applied to the moved interface, just use normal networking configuration in /etc/rc.conf inside the jail. This could be fixed with a patch too, though, just needs some time which is in short supply. After the jail comes up, you'll see an unconfigured vnet0 interface which you can ignore.

The (usually) above is just a caveat of vnet mode in general; when you restart the jail (using 'iocage restart') the vnet stack is not always properly torn down, and as a result the interface you moved into the jail doesn't get moved back to the host... so it's gone. The only fix for this situation is to reboot the machine unfortunately.

 

[diskdiddler]

Can anyone confirm if this problem persists? I've just logged a bug, because even under BETA1, my networking is entirely, suddenly dead for iocage.

Frustratingly, it WAS working fine on U5, now dead on BETA1 as well. Just stopped, update didn't fix?

 

[j0hnby]

I still have to have the Tunables in to have the physical NIC added into the bridge. Although at the bridge does get created on it's own now. I thought it had been fixed in 11.2B1, but might have been reading the U6 info instead and presumed....fail

 

[spotcatbug]

Can anyone tell me what my tunables should look like if my network connection is an aggregation? The interface is named "lagg0". The problem is am finding is that if I replace the "igb0" above with "lagg0", I am unable to access the entire freenas system after reboot (until i reconfigure the interface from the local kb/monitor).

Finally! Someone with the same problem I have! I have warden and iocage jails and an aggregated network connection.

After much fiddling I now have a working "solution." Since I rarely need to restart my FreeNAS machine, it's all manual. It could be scripted, but I'm just doing it manually whenever I have to restart FreeNAS because it's all going to get sorted-out eventually, right?

Anyway, steps:

1. Set your warden jails to not autostart.
2. Reboot.
3. Open a shell and do ifconfig -a. Notice your iocage jail's interface and a bridge interface. The bridge interface will have your iocage interface in it; however, lagg0 won't be in it, so...
4. sudo ifconfig bridge0 addm lagg0 (at this point your iocage jail will work.)
5. Now start each of your warden jails (at this point your warden jails will work.)
6. ifconfig -a. Notice that all the jail interfaces are in bridge0 along with lagg0.

You have to do steps 4 and 5 whenever you reboot your FreeNAS machine.

 

[Ceetan]

Code:

root@freenas-pmh:~ # cat /mnt/zfs/scripts/bridge0.sh
#! /bin/sh

PAUSE="10"
BRIDGE="bridge0"
INTERFACES="igb1 igb2 igb3"

(
	until /sbin/ifconfig | /usr/bin/grep -q "^${BRIDGE}:"; do sleep 2; done
	sleep ${PAUSE}

	for if in ${INTERFACES}
	do
		/sbin/ifconfig ${if} up
		/sbin/ifconfig ${BRIDGE} addm ${if}
	done
) &

exit 0

@Patrick M. Hausen Just to make sure I understand: This is used as a post init-scrip in default freeenas userspace?

What does the first line (starting with cat) do?

 

[Linkman]

@Patrick M. Hausen Just to make sure I understand: This is used as a post init-scrip in default freeenas userspace?

What does the first line (starting with cat) do?

The 'cat' command is what is echoing the contents of 'bridge0.sh' to the terminal. The actual script starts with '#! /bin/sh' and ends with the line 'exit 0'

 

[nathank1989]

I am running 11.2-BETA1 and my iocage jails are wrecking havok on my NAS connectivity and my network as a whole.
My Ubiquiti UniFi system will start losing access to IP ranges because of some confusion from the NAS. I cannot reliably keep access to my jails, setting static specific interfaces loses my access entirely.

This is maddening since I know Warden jails worked for my use case, but want to start fresh on iocage so as not to worry about upgrading later.... but here I am pulling my hair out trying to achieve a basic goal.

 

[Jailer]

I am running 11.2-BETA1 and my iocage jails are wrecking havok on my NAS connectivity and my network as a whole.

Try Beta 2 to see if that helps, it just dropped.

https://download.freenas.org/11.2/STABLE/BETA2/x64/

 

[Patrick M. Hausen]

@Patrick M. Hausen Just to make sure I understand: This is used as a post init-scrip in default freeenas userspace?

Yes it is - put it anywhere on your ZFS and use the Web UI to set is as a post-init task.

What does the first line (starting with cat) do?

Thanks Nathan for answering that one - see above.

Patrick

 

[Zwck]

how do you guys create your jails at the moment, in terms of network settings (Beta2)?

 

[verysneaky]

I just upgraded to 11.1-U6 to fix this issue, but it's still persisting. Anybody else having the same problem or am I doing something wrong?

 

[j0hnby]

how do you guys create your jails at the moment, in terms of network settings (Beta2)?

Used to do it via the command line, but since B2 have used the web interface - it's much more slick. Still prefer to create a blank jail then add my own packages in manually, that's just me liking to know how things go together though.