iocage jails in VLANs

[alexr]

Using 11.2 and trying to build iocage jails in their own VLANs. I'm trying to stay on the new GUI path as much as possible.

The primary interface is lagg0. There's a VLAN interface, vlan20, with lagg0 as it's parent. The new jail is set for DHCP.

iocage start foo fails because it couldn't obtain a DHCP address. It then auto-stops the jail, leaking the interfaces it creates <https://github.com/iocage/iocage/issues/757>.

I've gathered tcpdumps from the firewall (DHCP Server), on lagg0, and on vlan20. All of them show the DHCP DISCOVER with the proper VLAN tag. The vlan20 interface is the only place where the DHCP OFFER doesn't appear, and it has the correct VLAN tag on it.

What am I missing?

 

[alexr]

Hmm, iocage’s docs recommend setting net.inet.ip.forwarding and a few other sysctls, but they're not set by the GUI. I’d expect that when setting up a vnet jail in the GUI that it would handle this sort of thing automatically.

Is this unnecessary because of the bridge that the GUI is setting up?

 

[alexr]

Ok, so setting the sysctls recommended by the iocage docs do not help. I did another set of tcpdumps and included the iocage-generated bridge as well this time and it's not seeing the VLAN traffic, despite having vlan20 as a member.

 

[dlavigne]

Does this comment help? https://redmine.ixsystems.com/issues/62268#note-2

 

[alexr]

Thanks, Dru.

I already have vlan_default_interface=vlan20. The lagg0 interface isn't in the iocage-created bridge1. It can't even be added manually because it returns EBUSY if you try. I presume this is because it's the parent of vlan20 and thus in the span list.

By Allan Jude's suggestion, I tried "dhclient vlan20" on the host and that does see the firewall's OFFER. I'd prefer not to have the host on the vlan -- we're building a unique vlan for each external service to minimize attack surface area.

Here's what things look like while the jail is started:

Code:

ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether XX:XX:XX:XX:XX:48
    hwaddr XX:XX:XX:XX:XX:48
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether XX:XX:XX:XX:XX:48
    hwaddr XX:XX:XX:XX:XX:49
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether XX:XX:XX:XX:XX:48
    inet6 fe80::XXXX:XXXX:XXXX:XX48%lagg0 prefixlen 64 scopeid 0x4
    inet 192.168.0.245 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether XX:XX:XX:XX:XX:48
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 20 vlanpcp: 0 parent interface: lagg0
    groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:10:4a:a8:b1:00
    inet YY.YYY.YY.YY netmask 0xff000000 broadcast YY.255.255.255
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 1000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: dev
    options=8<VLAN_MTU>
    ether 02:ff:60:6f:45:b3
    hwaddr 02:a7:50:00:07:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:10:4a:a8:b1:01
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:19 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 1000
vnet0:19: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: foo
    options=8<VLAN_MTU>
    ether 02:ff:60:37:94:2f
    hwaddr 02:a7:50:00:09:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

 

[alexr]

For those following along, a few potential clues I dug up:

I have a similar motherboard to the below (I'm X11DPH-Tq and below is X11DPH-T):

https://forums.freenas.org/index.ph...iver-instead-of-ixgbe-for-10g-lan-card.57930/ and https://redmine.ixsystems.com/issues/25937

and then this:

ixl problems on FreeBSD (XL710)

 

[sotiris.bos]

For those following along, a few potential clues I dug up:

I have a similar motherboard to the below (I'm X11DPH-Tq and below is X11DPH-T):

https://forums.freenas.org/index.ph...iver-instead-of-ixgbe-for-10g-lan-card.57930/ and https://redmine.ixsystems.com/issues/25937

and then this:

ixl problems on FreeBSD (XL710)

Hey, I have been reading your posts to try and figure out how to fix my problem that kinda looks like yours, without the lagg part.

It seems to me that jails cannot accept DHCP offers on FreeNAS Interfaces that are not tagged as DHCP in the UI under the Network, Interfaces menu.

Check out my post as well: https://forums.freenas.org/index.php?threads/cant-get-jails-to-work-with-vlan.72776/

 

[alexr]

Destroying and then building up the entire interface hierarchy by hand works (thanks for the suggestion, Caleb).

That means the problem is in how rc/middleware brings up the interfaces.

There's an odd message in logs:

Code:

Jan 19 18:38:01 freenas kernel: lagg0: link state changed to UP
Jan 19 18:38:01 freenas kernel: lagg0: link state changed to UP
Jan 19 18:38:01 freenas ixl1: ixl_init_locked: reconfigure MAC addr
Jan 19 18:38:01 freenas ixl0: aq_add_macvlan err -53, aq_error 14

This message appears to be the Intel driver trying to setup something about the vlan interface before something is ready for that in the NVM firmware.

/tmp/rc.conf.freenas has this:

Code:

ifconfig_lagg0="DHCP laggproto lacp laggport ixl0 laggport ixl1"

ifconfig_lagg0_ipv6="inet6 accept_rtadv auto_linklocal"
rtsold_enable="YES"
cloned_interfaces=" lagg0"
ifconfig_vlan20=" up"

ifconfig_ixl0=" up"
ifconfig_ixl1=" up"
vlans_lagg0="vlan20"
create_args_vlan20="vlan 20 vlanpcp 0"

This looks a bit wrong to me.

 

[alexr]

OK, I've reproduced this with a vanilla FreeBSD 12 config:

1. zpool remove <log> # sort out the NVDIMM
2. zpool remove <cache> # prevent pool confusion later from test OS
3. shutdown and remove all hard drives
4. Boot from FreeBSD 12.0 DVD .ISO and install to a fresh USB key.
5. Change default rc.conf to have:
ifconfig_ixl0="up"
ifconfig_ixl1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto lacp laggport ixl0 laggport ixl1 DHCP"
6. Exit installer and boot off USB key.
7. ssh in via lagg0 and immediately su.
8. Add the vlan, bridge, and epair:
root@fb12:/usr/home/alexr # ifconfig vlan20 create vlan 20 vlandev lagg0
root@fb12:/usr/home/alexr # ifconfig bridge20 create addm vlan20
root@fb12:/usr/home/alexr # ifconfig epair0 create up
epair0a
root@fb12:/usr/home/alexr # ifconfig bridge20 addm epair0a
9. dhclient epair0b

No OFFER packets make it through the bridge to the epair, but the packets are seen on a tcpdump of lagg0. Again, destroying the epair, bridge, vlan interfaces and then recreating them after this point works as expected.

 

[ezra]

Any update? Same situation... pulling my hair out!

 

[sotiris.bos]

Any update? Same situation... pulling my hair out!

Are you using a lagg interface or not?

 

[ezra]

Stepped away from lagg in order to first fix this (lagg gave additional trouble, not persistent on reboot)
I'm using a switch vlan right now with a dedicated nic for each vlan (4x), rather have it setup via freenas on the lagg interface...

 

[sotiris.bos]

Stepped away from lagg in order to first fix this (lagg gave additional trouble, not persistent on reboot)
I'm using a switch vlan right now with a dedicated nic for each vlan (4x), rather have it setup via freenas on the lagg interface...

Yes, that is what I resorted to as well and what I was going to suggest. Untagged VLANs on the switch with FreeNAS interfaces for each VLAN is the way to go for the time being.

 

[ezra]

Yeah, just to bad, i spend about a full month on this issue... and i have 8 NIC's on my system ^^
Well i'll keep anyone updated if i find a solution, hope you guys will do the same.

 

[alexr]

Still patiently awaiting a response from the ix engineer who said a few weeks ago that he'd look into it right away.

 

[dlavigne]

Which ticket number?

 

[alexr]

Which ticket number?

@dlavigne 70419

 

[ezra]

@dlavigne 70419

https://redmine.ixsystems.com/issues/70419
403 not found, could you provide a link, id like to chime in.

 

[alexr]

https://redmine.ixsystems.com/issues/70419
403 not found, could you provide a link, id like to chime in.

It’s currently private due to log data.

If you post here, I can forward into the issue.

 

[ezra]

I'd be glad, i got it working! Have to test the leaking though.

https://github.com/iocage/iocage/issues/873
last post

Could you point me in a direction to test the leakages (though they all start, cant ping other subnets etc) with tcp dump? I want to be assured that nothing leaks.