iocage Jails, Multiple NICs, Different Subnets, Best Practice

[nathank1989]

I have begun using 11.2-BETA2 on a home system.
I am using an ASRock Rack D1540d4i which comes with two onboard gigabit NICS.

I have a UniFi network stack with 3 vlans, 10.0.0.0/24 (CORE); 10.10.0.0/24 (LAN); 10.20.0.0/24 (SOHO)

I have read https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/
and https://forums.freenas.org/index.php?threads/lacp-friend-or-foe.30541/

and decided against agg because I will have less than 10 busy clients, and want jails on one subnet while mgmt and file serv is on another.

I want the management IP to be on CORE, and accessible from all my vlans, I also want my NFS and SMB shares off of this vlan through the igb0 interface

I want two iocage jails to go through igb1 interface on the SOHO vland accessible from other vlans.

I went into this thinking it was simpler than it is, and I don't know where to begin to get this to work correctly. I have tried manually adding vlans, interfaces, setting jails static, I just can't get the scenario I want to happen and I am beginning to question whether it is best practice.

 

[Chris Moore]

I have a UniFi network stack with 3 vlans, 10.0.0.0/24 (CORE); 10.10.0.0/24 (LAN); 10.20.0.0/24 (SOHO)

Routing between vlans is something that happens at the router or in a switch that is able to handle the task. It is a setting outside of FreeNAS.

I am beginning to question whether it is best practice.

Where did you get the idea that it was a best practice? In a large organization, like where I work, we use vlans to separate the systems, one department from another, and also set different IP ranges for systems on different vlans. That way, they can't talk to each other unless we want them to. For example, all the printers are on a separate vlan that has no internet access to keep the network attached printers from talking to the mother ship. There are times when vlans are needed, but I would not do it at home unless I was just trying to learn about how it works. It is a lot of trouble to setup. If you are making a typical home NAS, or even a small business, it is probably not needed.

However, there might be something easier to setup that would serve the purpose you are trying to acomplish. Leaving the vlan idea to the side for now, what is it you are really trying to accomplish in this scenario?

 

[nathank1989]

Routing between vlans is something that happens at the router or in a switch that is able to handle the task. It is a setting outside of FreeNAS.

Where did you get the idea that it was a best practice? In a large organization, like where I work, we use vlans to separate the systems, one department from another, and also set different IP ranges for systems on different vlans. That way, they can't talk to each other unless we want them to. For example, all the printers are on a separate vlan that has no internet access to keep the network attached printers from talking to the mother ship. There are times when vlans are needed, but I would not do it at home unless I was just trying to learn about how it works. It is a lot of trouble to setup. If you are making a typical home NAS, or even a small business, it is probably not needed.

However, there might be something easier to setup that would serve the purpose you are trying to acomplish. Leaving the vlan idea to the side for now, what is it you are really trying to accomplish in this scenario?

I am doing this as a learning experiment for larger enterprise networks scaled down to two dozen machines.

I'd like my NAS Management and local storage access (NFS, SMB) to go through igb0 on the main CORE Lan (10.0.0.0/24) but accessible from all vlans. Right now my UniFi set up does not block inter-vlan communication as I intended.
I'd like my Jails to go through igb1 on VLAN 20 (10.20.0.0/24)

 

[Chris Moore]

I am doing this as a learning experiment for larger enterprise networks scaled down to two dozen machines.

I'd like my NAS Management and local storage access (NFS, SMB) to go through igb0 on the main CORE Lan (10.0.0.0/24) but accessible from all vlans. Right now my UniFi set up does not block inter-vlan communication as I intended.
I'd like my Jails to go through igb1 on VLAN 20 (10.20.0.0/24)

I am afraid that you are mixing up two different things. Vlan tagging and IP subnetting are two different things, although they are often used in conjunction with one another. If you have a misunderstanding of what you are doing, you may be creating your own pain.
Take a look at these two references and see if they don't shed some light on your difficulty:
Re: IP Subnetting
https://support.microsoft.com/en-us...nding-tcp-ip-addressing-and-subnetting-basics

Re: Vlan tagging
http://www.firewall.cx/networking-topics/vlan-networks/219-vlan-tagging.html

They actually have a lot of good documents: http://www.firewall.cx/networking-topics.html

 

[nathank1989]

I will give thos a read, but all I want to do is connect my NAS to two ports on my UniFi switch and one port on one IP range and the other port on a different IP range and have them still talk to each other. Seeing as how two NICS on same subnet is a no-no, and link aggregation is only worth it for heavy loads, I am stuck as to what it is I need to do to get management on one NIC and Jails on the other

 

[short-stack]

I will give thos a read, but all I want to do is connect my NAS to two ports on my UniFi switch and one port on one IP range and the other port on a different IP range and have them still talk to each other. Seeing as how two NICS on same subnet is a no-no, and link aggregation is only worth it for heavy loads, I am stuck as to what it is I need to do to get management on one NIC and Jails on the other

I operate a similar setup at my house with a USG and Unifi switches and multiple VLANS with iocage jails on the different VLANs and restrictions in between.

I work from home and my wife occasionally does as well. I also do not trust most IoT devices, nor the company my wife works for to properly manage her laptop so I have a mangement VLAN which is the native VLAN in my Unifi setup, and then I have a VLAN for my Home devices, one for IoT, one for my work, one for my wife's work (that only allows outbound internet access, and one for a guest wireless network.

Of all of those VLANs, I have a 2 port LAGG to my FreeNAS box but with only the management 172.30.20.0/24 as the native, and then with my work 192.168.252.0/24, and my home network 192.168.254.0/24 on the trunk, and the IoT/DMZ 192.168.250.0/24.

First place to start is the network. Create your VLANs, set up your firewall rules, get everything working how it should before you add the FreeNAS box to the mix.
Once traffic is flowing/blocking as expected, go ahead and set up your LAGG with a port profile that specifies your native/tagged traffic.

Then in FreeNAS create the LAGG, add the interface members, and give it an IP on the management network since that is your native VLAN.

After that you're ready to create VLAN interfaces for each of the networks you are trunking to FreeNAS, give them all the parent interface of the LAGG, and then an IP address on their respective VLANs. I used .5 for each network, so no matter which network I was on I could hit x.x.x.5 and get to the box.

That's all that is needed. I got a little crazier from there though, I run a few IDS's on a VM and have a 3rd NIC on FreeNAS that accepts SPAN traffic from my network that is analyzed.
So that VM has one NIC on VLAN 252 and one NIC that's the physical em2 NIC.

I also have jails all over, I run the Unifi controller in a jail on the management network, run Plex in a jail on the home network, and run a few random test jails/vms on my work network.

It's all doable, but you have to have the network working before you can introduce the FreeNAS box.

Good luck! I can answer any questions you might run in to.

 

[nathank1989]

I operate a similar setup at my house with a USG and Unifi switches and multiple VLANS with iocage jails on the different VLANs and restrictions in between.

I work from home and my wife occasionally does as well. I also do not trust most IoT devices, nor the company my wife works for to properly manage her laptop so I have a mangement VLAN which is the native VLAN in my Unifi setup, and then I have a VLAN for my Home devices, one for IoT, one for my work, one for my wife's work (that only allows outbound internet access, and one for a guest wireless network.

Of all of those VLANs, I have a 2 port LAGG to my FreeNAS box but with only the management 172.30.20.0/24 as the native, and then with my work 192.168.252.0/24, and my home network 192.168.254.0/24 on the trunk, and the IoT/DMZ 192.168.250.0/24.

First place to start is the network. Create your VLANs, set up your firewall rules, get everything working how it should before you add the FreeNAS box to the mix.
Once traffic is flowing/blocking as expected, go ahead and set up your LAGG with a port profile that specifies your native/tagged traffic.

Then in FreeNAS create the LAGG, add the interface members, and give it an IP on the management network since that is your native VLAN.

After that you're ready to create VLAN interfaces for each of the networks you are trunking to FreeNAS, give them all the parent interface of the LAGG, and then an IP address on their respective VLANs. I used .5 for each network, so no matter which network I was on I could hit x.x.x.5 and get to the box.

That's all that is needed. I got a little crazier from there though, I run a few IDS's on a VM and have a 3rd NIC on FreeNAS that accepts SPAN traffic from my network that is analyzed.
So that VM has one NIC on VLAN 252 and one NIC that's the physical em2 NIC.

I also have jails all over, I run the Unifi controller in a jail on the management network, run Plex in a jail on the home network, and run a few random test jails/vms on my work network.

It's all doable, but you have to have the network working before you can introduce the FreeNAS box.

Good luck! I can answer any questions you might run in to.

This is extraordinarily helpful! Thank you I will give this a shot.

 

[StarkJohan]

Did you accomplish what you wanted?

I'm running a similar setup and with every update I've had issues. Upgraded to 11.2 beta 2 and had to recreate the jail from scratch to get it working again. Multiple nics in the FreeNAS machine. igb0 is the main NIC where most traffic passes through. No specified VLANs on igb0. The second NIC igb1 is connected to a port on my switch which is just Untagged VLAN4 and I would like a specific jail to use igb1 exclusively.

Default vlan is untagged and uses default gateway 10.10.10.1. vlan4 uses gateway 10.10.40.1.

I've tried a bunch of different ways of doing this since iocage was introduced but the only way I had success is using vnet in the jail because as far as I understand it this is required for the iocage "defaultrouter" and "resolver" settings to be considered. When starting the jail bridge0 and vnet0 is created on the host. Bridge0 has igb0 and vnet0 as members which of course is not correct. I then have to manually modify the bridge to contain the correct members, vnet0+igb1, but absolutely not igb0 as it does by default. The jail has a static ip set 10.10.40.40, defaultrouter 10.10.40.1 and resolver "nameserver 10.10.40.1". This works, but if I should restart the jail networking is screwed up as the bridge is recreated with the wrong members. Physical access to the machine is then the only way of fixing it which is rather annoying.

I wish I could manage the same setup without using vnet. If anyone managed to tag a jail to use a specific VLAN or a just use a specific NIC I would be very interested to know how that is set up and if it "just works". I currently have a postinit script running to fix the bridge to match my needs.

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

ether 02:cf:68:30:cb:00

nd6 options=1<PERFORMNUD>

groups: bridge 

id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15

maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200

root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

		ifmaxaddr 0 port 2 priority 128 path cost 20000

member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

		ifmaxaddr 0 port 5 priority 128 path cost 2000

vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

description: associated with jail: Torrent

options=8<VLAN_MTU>

ether 02:ff:60:09:44:62

hwaddr 02:2b:d0:00:05:0a

nd6 options=1<PERFORMNUD>

media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

status: active

groups: epair 

 

[short-stack]

If anyone managed to tag a jail to use a specific VLAN or a just use a specific NIC I would be very interested to know how that is set up and if it "just works".

I’ve set jails to both VLANs and interfaces.

You just have to specify it in the up address and the interfaces.

For a jail using a VNET in a VLAN set the up address to vnet0 and then assign the VNET to the right bridge for that VLAN

Code:

ip4_addr:vnet0|192.168.254.2/24
interfaces:vnet0:bridge0

For a jail not using VNET put the VLAN in the IP address. If you’re not using VNET you don’t have to specify the bridge in the interface, because it basically just creates a virtual IP on top of the VLAN, but it would look like this.

Code:

 ip4_addr:vlan250|192.168.250.10/24
interfaces:vnet0:bridge0

Doing a specific interface would be the same as either of the above depending if you want to use VNET or not, just put em1 or ixgb0 or whatever the interface name is in instead of vnet0 or vlan250 in the IP address field.

Edit: please note that for the current 11.2-BETA2 when you start a VNET enabled jail there is a bug that adds the LAGG interface to the bridge which breaks networking for anything using that bridge. I’m not sure if this happens if aren’t using a LAGG, but the manual workaround is to run

Code:

ifconfig bridge0 deletem lagg0

 

[nathank1989]

Yea, I just gave up and created a lagg interface and vlan'd the jails and now it works mostly how I want it to.

 

[StarkJohan]

Doing a specific interface would be the same as either of the above depending if you want to use VNET or not, just put em1 or ixgb0 or whatever the interface name is in instead of vnet0 or vlan250 in the IP address field.

Have you tried this using a different default gateway? I need the different NICs to be on separate subnets with separate default gateways. Considering that, your suggestion does not work.

I would also be alright with a solution where the jail traffic was sent on a separate NIC with a separate VLAN tag but this not seem to work either on account of the default gateway problem.

 

[short-stack]

I need the different NICs to be on separate subnets with separate default gateways. Considering that, your suggestion does not work.

That’s kinda the whole point of VLANs. You should only have one broadcast domain per network. Each VLAN should have its own subnet, and gateway.


Sent from my iPhone using Tapatalk

 

[StarkJohan]

That’s kinda the whole point of VLANs. You should only have one broadcast domain per network. Each VLAN should have its own subnet, and gateway.

Exactly. But my understanding and experience is that an iocage jail needs vnet to be ON to accept the "defauttrouter" and "resolver" settings, right?
If I create a jail with for example "ip4_addr:<secondary nic>|<ip in secondary subnet>/24" the jail does not find the "secondary subnet" gateway and thus is not able to do much networking at all.

 

[short-stack]

Exactly. But my understanding and experience is that an iocage jail needs vnet to be ON to accept the "defauttrouter" and "resolver" settings, right?
If I create a jail with for example "ip4_addr:<secondary nic>|<IP in secondary subnet>/24" the jail does not find the "secondary subnet" gateway and thus is not able to do much networking at all.

No. VNET being on basically just virtualized the jails network stack and the only reason it’s needed is if the jail runs something (Plex) that needs its own MAC address because of the way it functions.

A VLAN’s network can be set to anything, it’s not tied to FreeNAS.

FreeNAS networks the way it should, if you put a jail/VM on a VLAN it tags the traffic and sends it out to the switch/router to be passed along from there. FreeNAS doesn’t do the heavy lifting of VLANs/ advanced networking, the networking hardware does.


Sent from my iPhone using Tapatalk

 

[StarkJohan]

I think we're misunderstanding each other... default router only applies to vnet jails according to the iocage manpage:

Code:

defaultrouter=none | ipaddress
Setting this property to anything other than none configures a
default route inside a VNET jail.

I've had no luck since 9.3 all through to 11.0 with putting a jail on a vlan so I haven't really tried it recently. That wouldn't really accomplish what I'm going for either as that would send the traffic on the default freenas Nic, wouldn't it?