Jail not tagging packets with VLAN tag appropriately

[short-stack]

I recently switched out my firewall and started putting more restrictive rules outbound in place and I noticed that some of my Jail traffic was coming in to my firewall on the wrong interface. I have several VLANs that connect from firewall to switch and from switch to NAS over carrying several VLANs.

After some troubleshooting to identify where problem was being initiated, it's only on Jails that don't utilize VNET. All of my virtual machines, and any jail that has VNET are tagged appropriately, but a standard jail that is configured on a specific VLAN is sending traffic out tagged with VLAN 1,

It worked for years, but I wasn't running anti-spoofing or restrictive outbound rules on the firewall so it seemingly worked itself out coming in to the firewall on the native VLAN and going back out on the VLAN interface.

I monitored a SPAN port on the switch where my NAS is connected and took a packet capture to look at how the VLAN tagging was being applied.

The screenshots below show a TCP connection where I did a simple curl of google.com on a standard jail, a jail with VNET, and a VM.

On the standard jail, the packets are tagged VLAN1 on the way out, but you can see VLAN 250 on the way back in.

This is a jail on VLAN 254 that uses VNET that is being tagged appropriately.

This is a VM on VLAN 250 that is being tagged appropriately.

If anyone has any ideas, or if I missed a configuration item I'd being interested to hear them.

Here are the basic and network settings for the jail in question:

 

[dak180]

Here are the basic and network settings for the jail in question:

Can you also list the vlan definitions in the main system and post the output of the following commands: iocage get allow_raw_sockets <jail name>, iocage get interfaces <jail name>, iocage get resolver <jail name>, iocage get vnet_default_interface <jail name>, iocage get ip4_addr <jail name> and ifconfig?

 

[short-stack]

Can you also list the vlan definitions in the main system and post the output of the following commands: iocage get allow_raw_sockets <jail name>, iocage get interfaces <jail name>, iocage get resolver <jail name>, iocage get vnet_default_interface <jail name>, iocage get ip4_addr <jail name> and ifconfig?

The VLAN and interfaces are set up properly, all other traffic flows exactly how it should. This only happens specifically in jails without vnet enabled.

This is VLAN 250's definition

Description:​

DMZ

Active Media Type:​

N/A

Active Media Subtype:​

N/A

VLAN Tag:​

250

VLAN Parent Interface:​

lagg0

Bridge Members:​

N/A

LAGG Ports:​

N/A

LAGG Protocol:​

N/A

MAC Address:​

40:a8:f0:26:08:b8

MTU:​

N/A

Jail settings:

allow_raw_sockets:1
interfaces:vnet0:bridge1
ip4_addr:vlan250|192.168.250.10/24
resolver:nameserver 1.1.1.1; nameserver 8.8.8.8
vnet_default_interface:vlan250

ifconfig for lagg0 and vlan250:

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Primary LAG
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 40:a8:f0:26:08:b8
inet 172.30.20.5 netmask 0xffffff00 broadcast 172.30.20.255
laggproto lacp lagghash l2,l3,l4
laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>


bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:45:7a:e0:85:01
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 21 priority 128 path cost 2000
member: vnet2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 18 priority 128 path cost 2000000
member: vlan250 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>


vlan250: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: DMZ
options=80001<RXCSUM,LINKSTATE>
ether 40:a8:f0:26:08:b8
inet 192.168.250.5 netmask 0xffffff00 broadcast 192.168.250.255
inet 192.168.250.10 netmask 0xffffff00 broadcast 192.168.250.255
groups: vlan
vlan: 250 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

 

[dak180]

This is VLAN 250's definition

All fields from Network > Interfaces > Edit please.

Jail settings:

Seems ok, but need the preceding info to be sure.

Also, do you (on the main system) have any static routes set and if so what are they?

 

[short-stack]

All fields from Network > Interfaces > Edit please.


Seems ok, but need the preceding info to be sure.

Also, do you (on the main system) have any static routes set and if so what are they?

Again, it's not the interface or the VLAN or static route configuration. Many components operate as intended, it's only jails without vnet that don't tag the traffic correctly, but here are the requested settings.

 

[Patrick M. Hausen]

I assume you have assigned the jails IP address to the VLAN interface in question on the host, if the jail is non-VNET. Where is the system contacting the jail located? Same VLAN? If not, you are falling for the same assumption that pops up in this forum repeatedly.

Being: if a packet is sent with a particular source IP address it will leave the system over the interface that this address is assigned to. This is practically never the case if the other end of the conversation is not directly or indirectly (route) connected to that same interface. If it isn't, the packets leave via the default route and that is that.

This is not a shortcoming of TrueNAS but how every Unix based host on this planet works.

TrueNAS is not a complete network virtualisation on its own.

On the plus side, VMs and VNET jails do come with their own completely separate IP stack, so if you can switch your jail to VNET, I recommend doing just that, and the packets will go out the VLAN.

You should

• Statically create the bridge interface matching the VLAN with the VLAN as only member.
• Assign all host IP addresses that might be necessary to the bridge and not the VLAN interface.
• Assign the bridge interface to the jail's vnet0 in the network settings.

HTH,
Patrick

 

[short-stack]

Being: if a packet is sent with a particular source IP address it will leave the system over the interface that this address is assigned to. This is practically never the case if the other end of the conversation is not directly or indirectly (route) connected to that same interface. If it isn't, the packets leave via the default route and that is that.

I just changed the system default gateway and this seems to be the case.

I have to ask why there is an option in the jail settings for a default router if it's not used then? The 'same assumption' of setting a default gateway in the jail and having it be used as a default gateway is generally a fairly safe assumption to make.

 

[Patrick M. Hausen]

The separate default gateway of a jail works for VNET jails only. Non-VNET jails use the host networking. That's documented. That the UI permits filling the defaultrouter field when VNET is not active might warrant a bug ticket in JIRA, but is definitely a UI glitch and not a fundamental defect.

The only network related thing you can configure for a non-VNET jail is the nameserver setup, because that happens in the resolver library inside the jail. IP and everything related happens outside the jail in the kernel in the case of a jail without VNET.

 

[Patrick M. Hausen]

P.S. why are you running your jails without VNET at all? VNET is the greatest invention in the FreeBSD jail ecosystem since the invention of jails. I cannot think of any reason not to use it. I have more than 1000 VNET jails in my datacentre.

 

[short-stack]

The separate default gateway of a jail works for VNET jails only. Non-VNET jails use the host networking. That's documented

A valid IPv4 address to use as the default route. Enter none to configure the jail with no IPv4 default route. A jail without a default route will not be able to access any networks.

That’s the documentation for default router. Here: https://www.truenas.com/docs/core/applications/jails/create/

Every other field that’s VNET specific is explicitly marked as such, this is not.

As far as why not VNET, I wasn’t sure of the additional overhead and only utilized VNET where I needed to be able to utilize broadcast or MAC specific network functions.

 

[Patrick M. Hausen]

I was referring to the FreeBSD jail documentation. That's why I call it a UI defect. The networking does what it is supposed to do and that's not going to change. The TrueNAS frontend to that can use some improvement.

Believe me, this is just a small detail.

If you have a single network with a single interface and create a couple of VNET jails, TrueNAS automatically creates "bridge0" for you and assigns it to all the jails. While leaving the IP configuration of the physical interface unchanged.

Only problem: this configuration is explicitly forbidden. It's documented in FreeBSD. Member interfaces of bridge interfaces MUST NOT have an IP address configured. The IP address MUST go on the bridge interface.

I'm tired of ranting about that one ...

 

[dak180]

Patrick hit the nail on the head here. While there are other ways (not good ideas) to solve your issue the best way is to is to turn on vnet because TrueNAS is really meant to be a nas and not a router.

As far as why not VNET, I wasn’t sure of the additional overhead and only utilized VNET where I needed to be able to utilize broadcast or MAC specific network functions.

compared to overhead of running whatever setup is in the jail itself I would not worry about it.

If you have a single network with a single interface and create a couple of VNET jails, TrueNAS automatically creates "bridge0" for you and assigns it to all the jails. While leaving the IP configuration of the physical interface unchanged.

Only problem: this configuration is explicitly forbidden.

Do you happen to know if this has been reported upstream to iocage already?

 

[short-stack]

I was referring to the FreeBSD jail documentation.

Where in the FreeBSD handbook does it say anything about default routers for jails? I searched all of the doc before I posted this thread. There isn't a mention of it anywhere in the handbook itself here: https://docs.freebsd.org/en/books/handbook/jails/ or in the man page for jails here: https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=8&format=html

Even the iocage docs here https://iocage.readthedocs.io/en/latest/networking.html#shared-ip omit defaultrouter from the shared IP section, but there is no mention that it's only designated for VNET. So it can be interpreted but it isn't clear or specific.

Only problem: this configuration is explicitly forbidden. It's documented in FreeBSD.

The irony of this statement isn't lost on me. Directly above you tell me to read the FreeBSD documentation, but then here a few paragraphs down you mention how what is documented by FreeBSD is the opposite of how TrueNAS is configured to operate.

 

[jgreco]

I cannot think of any reason not to use it.

Because it adds unnecessary sucky complexity, obviously.

Hundreds of jails. Zero vnet in production.

vnet totally makes sense if you are doing "VPS" type services where you want a full featured FreeBSD jail that looks as much like an independent host as reasonably possible. That's great for VPS, but really a catastrophuque for merely isolating services.

We run OSPF for most production services. This allows us to build network topologies that are not married to conventional ethernet-oriented network design. By injecting a /32 route into the routing environment, and binding that to loopback (lo1), traffic finds its way to a server, and can even be floated around the network, without changing its IP address. Each server is connected into the network through at least two redundant paths.

We then bring up "apps", which are jails, simply by spinning up their IP address on lo1 and letting that be announced by the base platform. The jail inherits the routing behaviours of the base OS, which is correct and desirable. You really wouldn't want the jail doing anything else, particularly not doing its own routing, in such an environment. That would also break the ability of the base OS to do proper firewalling of the jail.

 

[jgreco]

The irony of this statement isn't lost on me. Directly above you tell me to read the FreeBSD documentation, but then here a few paragraphs down you mention how what is documented by FreeBSD is the opposite of how TrueNAS is configured to operate.

I think the problem you're really running into is that there are Just So Many Possible Ways For Things To Be Done. It is actually very difficult to come up with a cogent set of rules to explain all the options, advantages, caveats, and shouldn't-do's of networking.

Defaultrouters for jails aren't possible without vnet, and are "required" with vnet. Unless you use DHCP, in which case your vnet environment still picks up a defaultrouter, just provisioned via DHCP. People familiar with IP networking would consider this obvious; when you have an IP stack, it really "needs" a default route but cannot realistically have two (that's the "without vnet"). Adding vnet creates a second, virtual, network stack, so obviously then the host gets a defaultrouter and the virtual stack should also get one. But this is difficult to describe correctly in all its possible permutations, and there are other possible complications as well, such as the fact that a default route isn't ACTUALLY required by an IP stack, and in some cases, you might have a host OS without any defaultrouter, but a vnet jail WITH one, just for security purposes. Yikes-a-riffic.

And then the developers of FreeNAS/TrueNAS are trying to put that all behind a GUI. This is just ripe for exposing shortcomings from several angles. Not the least of which may be both documentation and coding related. So please give the developers a break, and do file bug reports where you run into problems.

Arguing with @Patrick M. Hausen about this isn't going to get you anywhere. I can get away with doing so (and I just did!) but I am fully aware of his networking experience, and you really are talking to someone here with significant experience and insights into the problem. Two people, actually, if we count me in as well. So I suggest that it may be more productive to see about reporting issues with the documentation and/or TrueNAS GUI code, iocage, etc.

 

[short-stack]

I think the problem you're really running into is that there are Just So Many Possible Ways For Things To Be Done. It is actually very difficult to come up with a cogent set of rules to explain all the options, advantages, caveats, and shouldn't-do's of networking.

Defaultrouters for jails aren't possible without vnet, and are "required" with vnet. Unless you use DHCP, in which case your vnet environment still picks up a defaultrouter, just provisioned via DHCP. People familiar with IP networking would consider this obvious; when you have an IP stack, it really "needs" a default route but cannot realistically have two (that's the "without vnet"). Adding vnet creates a second, virtual, network stack, so obviously then the host gets a defaultrouter and the virtual stack should also get one. But this is difficult to describe correctly in all its possible permutations, and there are other possible complications as well, such as the fact that a default route isn't ACTUALLY required by an IP stack, and in some cases, you might have a host OS without any defaultrouter, but a vnet jail WITH one, just for security purposes. Yikes-a-riffic.

And then the developers of FreeNAS/TrueNAS are trying to put that all behind a GUI. This is just ripe for exposing shortcomings from several angles. Not the least of which may be both documentation and coding related. So please give the developers a break, and do file bug reports where you run into problems.

Arguing with @Patrick M. Hausen about this isn't going to get you anywhere. I can get away with doing so (and I just did!) but I am fully aware of his networking experience, and you really are talking to someone here with significant experience and insights into the problem. Two people, actually, if we count me in as well. So I suggest that it may be more productive to see about reporting issues with the documentation and/or TrueNAS GUI code, iocage, etc.

I agree there are a thousand ways to do anything. I'm being more defensive that argumentative. I'm not that one that came in aggressively assuming a bunch of stuff that I had already described and documented in a post above, only to basically say RTFM and then later point out how the documentation sources often conflict.

That said, I searched the documentation and these forums and didn't find anything about traffic not being tagged with the 802.1Q tags correctly.

I do file bug reports, but only after I post and discuss something and identify that it's actually a bug and not mis-configuration or user error.

But when the help blurb is highlighted in bold that you won't be able to access anything without a default router, it can give the wrong impression and lead to people asking questions about how/why things are working the way they are.

 

[jgreco]

I searched the documentation and these forums and didn't find anything about traffic not being tagged with the 802.1Q tags correctly.

I think a lot of this goes to the definitions of "correctly". There are a lot of ideas out there, not always based in reality, about how networking and IP and vlans work, or are "supposed" to work. Some of this is complicated further by restrictions imposed within the OS that are not necessarily "desirable" from my point of view. Some of this stuff you just have to learn.

Unrelated to this specific issue, consider multiple interfaces on a single subnet.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

This totally doesn't work the way most people arrive here assuming that it does. Trying to educate people about what is actually going on is more productive than debating how someone feels it "should" work, which always seems to end with me:

Nobody in these forums really wants this stuff to be hard for other users to use, so it is best to see if we can find productive corrections to documentation or GUI. So I'm just encouraging trying to keep this productive. Since I don't actually use the jails or vnet stuff on TrueNAS, I have a limited amount of TrueNAS jail experience -- possibly less than yours! -- to work from, but I do have hundreds of running jails and extensive systems and networking design experience with the underlying bits.

Please trust that we're trying to get you, and this, straightened out. It's best for everyone not to get too defensive or ruffled. The realities here are admittedly suboptimal in a number of directions, but solutions are available. We can keep this headed in a positive direction.

 

[Patrick M. Hausen]

We then bring up "apps", which are jails, simply by spinning up their IP address on lo1 and letting that be announced by the base platform. The jail inherits the routing behaviours of the base OS, which is correct and desirable.

If your apps don't need their own lo0 to bind to, good for you.

 

[Patrick M. Hausen]

The irony of this statement isn't lost on me. Directly above you tell me to read the FreeBSD documentation, but then here a few paragraphs down you mention how what is documented by FreeBSD is the opposite of how TrueNAS is configured to operate.

That's because TrueNAS does it wrong and you need to manually configure your way around that instead of relying on the automatically created bridge. If you create the bridge interfaces you need and put all IP addresses on them, all is fine. The way TrueNAS does it kind of works, it "only" breaks multicast and IPv6. So most people who just fire up a plugin never notice.

I'll create an issue on JIRA - again. I don't know why the TrueNAS team ignores how bridges work in FreeBSD.

The documentation for that is in the bridge section of the FreeBSD handbook, because it is not specific to jails.

 

[Patrick M. Hausen]

Where in the FreeBSD handbook does it say anything about default routers for jails?

Sorry, my bad.

Virtualize Your Network on FreeBSD with VNET

Find out how to use the powers of VNET with FreeBSD jails and to isolate networks. Follow our guide and learn more about best practices with jails networking and how to get the most out of your VNET configuration.

klarasystems.com

This article among others makes it clear that only with VNET you get a separate network stack for the jail. And it is "obvious" that only with that you can have routing separate from the host. Non-VNET jails use the host's networking including all routes.