How to properly setup iocage jails using a VLAN

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Hi -- I've researched this topic for about the last few hours, however I'm slightly confused how to set things up

My setup pfsense router/ unifi switches and APs and 11.7 u7 FreeNAS on bare metal.

I've created all the vlans on pfsense and I have freenas connected to trunk port of Unifi switch passing all the VLAN tags.

bhyve jails are currently using vnet as network interface.

I created VLANs with pfsense GUI and tied them to the one physical interface coming into the machine igb0.
I'm not sure what to do beyond this.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
So I've managed to create the bridges and attach each VM to a bridge. Ie I created bridge30 for vlan30 and bridge0 for vlan1.
Within each jail I've set the IPV4 interface to vnet0 and interface to vnet0:bridge<vlan number>

All network packets entering FreeNAS on igb0 are tagged (including VLAN1 and VLAN30) tags.

However when starting the jail and attempting to ping another computer on the VLAN I keep getting host is down.

Code:
I have the following tunables set:
cloned_interfaces  bridge0 bridge30 rc i
fconfig_bridge0  addm vlan1 up  rc
ifconfig_bridge30  addm vlan30 up  rc
ifconfig_igb0  up  rc
net.add_addr_allfibs  0  sysctl
net.fibs  4  loader
route_vlan30_gw    default 10.0.30.1 -fib 3  rc
route_vlan30_if      -net 10.0.30.0/24 -iface vlan30 -fib 3  rc
route_vlan1_gw      default 10.0.1.1 -fib 1  rc
route_vlan1_if        -net 10.0.1.0/24 -iface vlan1 -fib 1   rc
static_routes  vlan1_if vlan1_gw vlan30_if vlan30_gw    rc
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
So I've managed to create the bridges and attach each VM to a bridge. Ie I created bridge30 for vlan30 and bridge0 for vlan1.
Within each jail I've set the IPV4 interface to vnet0 and interface to vnet0:bridge<vlan number>

All network packets entering FreeNAS on igb0 are tagged (including VLAN1 and VLAN30) tags.

However when starting the jail and attempting to ping another computer on the VLAN I keep getting host is down.
I think you may be making this more complicated than it needs to be; I have my jails working on vlans and have no tunables set for them.

Here is how I did that:
First in VLANs under Network set up your vlans (in my case and for this example 60).
Next Interfaces under Network set up a corresponding interface for each vlan ie: vlan60.
Then in the jail network properties set interfaces to vnet0:bridge60.
Finally in the jail network properties set vnet_default_interface to vlan60.


Change as appropriate for your vlans and everything should just work.
One nifty thing about this setup is that if you have two jails on the same vlan they will share a bridge which speeds up startup time.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Yea I managed to get things working they way you described with a few caveats.
I ended up tagging all traffic into FreeNAS including VLAN1.
I created bridges for VLAN1 and VLAN30 (in my case).
I added the VLANS to the appropriate bridges
I left vnet_default_interface at auto (and things still worked).
The problem I found out after much trial an error is I had a concurrent VM that I had created before that was bridged through bridge1 to the parent interface igb0. I actually needed to remove igb0 from the bridge -- and actually destroyed bridge1 and then moved the VM to connect to VLAN1 network. This last step actually took me about a day to figure out.

I didn't need any fibs statements (in fact they screwed things up as I found out as I went through the trial and error process since they bypassed the epair interfaces) and I didn't need to setup any static routes -- both these items I previously posted.
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
I left vnet_default_interface at auto (and things still worked).
The problem I found out after much trial an error is I had a concurrent VM that I had created before that was bridged through bridge1 to the parent interface igb0. I actually needed to remove igb0 from the bridge -- and actually destroyed bridge1 and then moved the VM to connect to VLAN1 network. This last step actually took me about a day to figure out.
One of the other nice things about setting things up the way I outlined is that there is no need for you to create the bridges yourself, all of that will happen automatically.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@dak180
I haven't test -- but what your telling me is if you specify something like you posted:

First in VLANs under Network set up your vlans (in my case and for this example 60).
Next Interfaces under Network set up a corresponding interface for each vlan ie: vlan60.
Then in the jail network properties set interfaces to vnet0:bridge60.
Finally in the jail network properties set vnet_default_interface to vlan60

then there is no need?? to set up system tunables like:

Code:
    cloned_interfaces    bridge0 bridge60    rc
    ifconfig_bridge0    addm vlan1 up        rc
    ifconfig_bridge30    addm vlan60 up        rc
    ifconfig_igb0        up                    rc
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
then there is no need?? to set up system tunables like:
Yes, exactly; I have exactly zero tunables that reference ifconfig or anything else with with my interface setup (plenty for other things though).

Doing that leads to something like:

Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether *****************
    hwaddr *****************
    inet ***************** netmask 0xffffff00 broadcast **************
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether *****************
    inet ***.***.60.*** netmask 0xffffff00 broadcast ***.***.60.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether *****************
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: vlan60 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 20000
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: *** as nic: epair0b
    options=8<VLAN_MTU>
    ether *****************
    hwaddr *****************
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0:3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: *** as nic: epair0b
    options=8<VLAN_MTU>
    ether *****************
    hwaddr *****************
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

And everything just works ☺️.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Hmm let me try setting up another VLAN using this method but not adding anything to the tunables to reflect the new VLAN -- ie like VLAN 50
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
upgraded to 11.3-rc2 -- still trying above
 
Last edited:

dak180

Patron
Joined
Nov 22, 2017
Messages
308
upgraded to 11.3-rc2 -- still trying above
11.3 changed the network interface management and as I have not yet upgraded, I make no guaranty that these directions will work as is on anything past 11.2.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Yea --- I'm thinking this isn't the case with 11.3 -- Honestly you have to setup the bridges manually via the GUI interface as I don't think they will be created automatically. It's much more informative BUUUTT at the same time it really screws things up
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@dak180

Ok so it looks like in 11.3 the Network->Interfaces section has changed. Within this section alone your can add bridges, vlans, LAGGs, etc.
There is no separate Network->VLAN section anymore since they basically combined this old section into Network->Interfaces.

Because of the incorporation of bridges into this interface, any bridges that were setup via system->tunables will be overwritten if they are referenced or required in the Network->Interfaces section.

The bridges will not be setup automatically within the Jails->Edit Tab.

Within Network->Interfaces you'll need to create your VLANs first, then create a bridge in the same section to add add the appropriate VLAN member (ie bridge30 - member VLAN30, bridge40 - member VLAN40). Add up for an option with each bridge since this will bring the bridge switch up.

At this point configure the Jail appropriate for the VLAN - making sure to add the vnet0:bridgex (x=appropriate VLAN number) under interfaces. If using static IP address make sure that networking all is in accordance to the VLAN network.

And finally FWIW,
I tagged all traffic entering my FreeNAS installation, so I have no untagged packets entering the box. I'm not sure if this is absolutely necessary, however I've seen some much older posts that recommended this strategy. I'm not sure if its still valid. This means I tagged VLAN1 at the exiting port of the switch that connects to FreeNAS. I also created a VLAN1 and associated it with a bridge so it could untag this VLAN1 traffic and present it to the system untagged.
 
Top