Security: Multiple Interfaces with Jails and VMs

[clysm]

tl;dr: Autostarting Jails and VMs assigned to separate physical interfaces causes everything to be bridged together.

I have a FreeNAS-11.2-U7 box with two network interfaces. The primary interface (igb0) is used for pretty much everything, including jails. This is on the "main" network. I would like to use the second interface (igb1) for serving a VM on the "management" network. However, I'm running into a security issue.

When all jails and VMs are set to autostart, FreeNAS will create a single bridge and place all physical interfaces used by these Jails/VMs within this bridge upon startup. This is a security issue. Broadcast traffic is leaking from one network to the other (main to management, and vice versa) and systems on the main network can access everything management network. Not good.

If I set the VM to not autostart (the only thing that should be using igb1), and later start the VM manually, FreeNAS will then create a second bridge for igb1. In this case, all traffic is isolated, but I have to log in and manually start the VM.

I cannot figure out how to isolate the physical networks so they are on separate bridges. Is there a tunable that am missing?



Note: Under my network interfaces, I only have igb0 added and set to DHCP. It makes no difference if I add igb1 or not.

Single bridge (all on autostart):
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:b0:9f:6d:66:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 55
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000

Dual bridge (autostart jails, manual start VM):
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:b0:9f:6d:66:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:b0:9f:6d:66:01
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 55
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000

 

[clysm]

The solution is somewhat simple: Specify the bridge for the Jails to be something other than "bridge0". To be extra safe, make it bridgeN or higher, where N is the number of physical network interfaces.

What I failed to mention previously is that the jails were configured with network interfaces of "vnet0:bridge0".

So, when everything is set to autostart, the VM appears to start first (maybe a race condition, but unfavorable in my case), and creates bridge0. When the jails start, they will see that bridge0 already exists and use it for their vnet adapters as well as the assigned physical interface. They don't check to see what is bound to it, they just use it.

If the jails start first, they will create bridge0 with vnets + igb0. When the VM starts later, it sees that bridge0 is in use and that it does not have the physical interface it needs. It will then create a new bridge and bind the tap and physical interface (igb1). Note that if another VM starts and wants to use the same physical interface, it will use the previously created bridge.

Manually configuring the bridge number for the Jails works, but seems delicate -- we have manually specified bridge numbers competing with auto-generated bridge numbers. Perhaps we should be able to specify a bridge number when configuring the VM interfaces as well.