SOLVED 9.3: multi-subnets / multi-gateways / multi-nics / multi-jails - FreeNAS routing question

[master-richie]

I have (successfully ) setup lacp and run vlan10 over it - the lag is NOT the default network / subnet and does not have an IP address assigned to it.

There are two jails using vlan10: 10.0.2.2 / 10.0.2.3 (default IP, vimage / NAT unchecked and nic set to vlan10) and both can be reached by other devices on the .2.x node. HOWEVER I cannot reach out to the internet from those jails.

Jail configuration tab shows the subnet the admin interface is on (10.0.1.x) and I cannot change it.

If I check vimage, I cannot choose my nic which I need to get them on vlan10

Do I need add a static route from shell for each jail?

 

[master-richie]

been working on this all morning. just changed the global setting default gateway to the gateway address for vlan10 (10.0.2.1) under network and BOOM - my vlan10 jails have internet access. this was after I ran a netstat -rn in one of the jails and saw the global default gateway address as the default gateway for the jail. apparently adding defaultrouter="10.0.2.1" in the /etc/rc.conf of the jails isn't enough. obviously this temp revelation ruins my FreeNAS admin internet access which is thru 10.0.1.x

I tried to use NO global gateway but then the jails admin page in the gui throws a warning about needing a default gateway and I cannot access the jails.

I'm really trying to setup a box with a couple of lacp lags passing vlans for various isolated purposes on isolated subnets (except plexmedia and owncloud which are same subnet / separate jails) - based on what I read this should work - am I barking up the wrong tree with this?

 

[Deleted47050]

Everything that you are seeing makes sense from a quick read. The default gateway is what ultimately gives you internet access, so the default gateway set to 10.0.1.1 will give internet access to your freenas box, while changing it to 10.0.2.1 will give your jails internet access instead, but you will lose internet access for your freenas system.

Using no global gateway obviously raises a warning as expected and, finally, your jails on the same vlan can ping each other without needing a default gateway since they are on the same subnet by definition.

You need to add a default gateway to your jails (10.0.2.1), but you also need to tell this gateway how to access the global gateway. Static routes would be a way to do this.

Edit: Ok, now that I am sitting in front of my computer I can add some details. The defaultrouter setting is not enough in the jails, as this specifies the default route for your jails, which is normally your default gateway. So this is good and necessary, but you are missing information on how to get to other subnets (in this case, your 10.0.1.x subnet).

Another idea, just throwing this out there, would be to have a FreeBSD instance acting as a gateway itself. You would configure multiple interfaces on it, and you would point both your FreeNAS system and your jails to the relevant gateway interface. This way, you could skip adding static routes as the FreeBSD gateway would already know the routes to all of the subnets as they are all directly connected to it already.

 

[master-richie]

Everything that you are seeing makes sense from a quick read. The default gateway is what ultimately gives you internet access, so the default gateway set to 10.0.1.1 will give internet access to your freenas box, while changing it to 10.0.2.1 will give your jails internet access instead, but you will lose internet access for your freenas system.

Using no global gateway obviously raises a warning as expected and, finally, your jails on the same vlan can ping each other without needing a default gateway since they are on the same subnet by definition.

You need to add a default gateway to your jails (10.0.2.1), but you also need to tell this gateway how to access the global gateway. Static routes would be a way to do this. Or you could also add a default route to 10.0.1.1.

well seeing as how my admin interface 10.0.1.x is a single physical connection and vlan10 comes in through the lag, I don't want to bridge my jails to route outbound traffic over the admin node - I want them to use the trunk their vlans are associated with so I cannot use static routes on the global interface.

Besides I just tried reverting the global default gateway to 10.0.1.1 AND adding a global static route of 0.0.0.0/24 -> 10.0.2.1 and it did nothing for the 2.x jails

or am I thinking of this wrong? should I just trunk all my interfaces into one big trunk, put an ip address on the trunk, run it as the admin node, pass my vlans over the trunk and then bridge them to the trunk's default gateway? of course if I do that then FreeNAS is acting as a "reverse" router, undoing the structure I have setup in my core router. vlan10 needs to stay on the 2.x subnet and NOT pass traffic over 1.x

I really need to keep 2.x traffic separated from 1.x (and others tbd)

 

[Deleted47050]

In this case, can't you just add a second virtual interface to your router for the 10.0.2.0 subnet?

Also, I don't think bridging should be necessary. You would simply add a static route inside your jails to 10.0.1.1, and a static route on your FreeNAS system to 10.0.2.1 (so that packets coming back from the internet can be correctly routed to the jails' subnet).

In your example, you have just added a default route to the 10.2.0.1 gateway (I had not clarified this enough in my initial post that you have quoted, edit followed), but there is nothing that tells your jails how to get to 10.0.1.1.

 

[master-richie]

In this case, can't you just add a second virtual interface to your router for the 10.0.2.0 subnet?

Also, I don't think bridging should be necessary. You would simply add a static route inside your jails to 10.0.1.1, and a static route on your FreeNAS system to 10.0.2.1 (so that packets coming back from the internet can be correctly routed to the jails' subnet).

In your example, you have just added a default route to the 10.2.0.1 gateway (I had not clarified this enough in my initial post that you have quoted, edit followed), but there is nothing that tells your jails how to get to 10.0.1.1.

I'm not sure what you mean with "adding a second virtual interface to my router" (because I have many vlans) and if you're telling me that I need to create a static route table on FreeNAS that tells vlan10 (10.0.2.x) how to get to the internet by going through the admin interface (10.0.1.x) then you are not understanding what I am asking.

1. router is setup where 10.0.1.x is the native vlan subnet and 10.0.2.x is vlan10
2. I'm using a Fortigate 60c router connected to the netgear switch via trunked line
   
3. netgear switch connects to FreeNAS box using 2 trunked lines and a normal ethernet line for admin purposes
4. the FreeNAS box has 7 NICS - 4 are setup as lagg0, 2 are setup as lagg1 (vlan10 / 10.0.2.x passes over this) and 1 is the embedded motherboard NIC which is used for FreeNAS admin (vlan1 / 10.0.1.x)
5. both my laggs are lacp
6. vlan1 is used for administration of all network devices - vlan10 is for my personal, family traffic: plexmedia, owncloud, etc
7. I am looking to pass vlan10 traffic to my two jails over the lagg vlan10 comes in on. vlan1 should not pass any vlan10 data.
8. should I be using VIMAGE and if so, how do I bind it to my vlan interface in FreeNAS?

with my current setup, is it possible for my two jails networking to be completely independent of the default admin adapter? can I achieve this with FreeNAS? this is what I am asking

 

[master-richie]

http://www.freebsdonline.com/content/view/742/524/

setfib

this would work perfectly ... has anyone tried this with FreeNAS or does FreeNAS use something similar? I'm now considering maybe I should wipe this machine clean and do a regular setup of FreeBSD 9.3, building a custom kernel

 

[Deleted47050]

I'm not sure what you mean with "adding a second virtual interface to my router" (because I have many vlans) and if you're telling me that I need to create a static route table on FreeNAS that tells vlan10 (10.0.2.x) how to get to the internet by going through the admin interface (10.0.1.x) then you are not understanding what I am asking.

This is not what I am saying. I am saying that if 10.0.1.1 is your border router, you need to tell your jails how to reach it, otherwise you are obviously going to lose internet access from your jails. This information is something that your jails cannot pick up for free, you need to specify this. One way to do this, as I said, would be to configure a static route. This way, packets from your jails would flow over VLAN10 (and the relative physical nics and cables), reach your VLAN10 gateway, and from there flow over the trunk to the border router.

This, by the way, does not mean that your jails will be "using your admin interface". You have configured trunking, so packets will be tagged with the correct VLAN number while going over the wire already and are thus kept separate from each other. What you need is a way to tell your router how to route a packet coming from the 10.0.2.0 network to the 10.0.1.1 gateway to access the internet. If the router is configured with an IP address on both networks, you would get this for free.

By adding a second interface I meant that you could configure your router with an interface on the 10.0.2.0 network, with IP address 10.2.0.1 acting as the default gateway for your VLAN 10. On Cisco equipment you would call this a subinterface. You would then set the default route of your jails to this IP address and you could avoid adding a static route to 10.0.1.1 because both networks would be connected for the router.

On a more practical level, just to try and clear some confusion here: what happens when you ping 10.0.2.1? Is that a machine? Is that an interface on the router? Do you get a reply when you ping this?

router is setup where 10.0.1.x is the native vlan subnet and 10.0.2.x is vlan10

1. I'm using a Fortigate 60c router connected to the netgear switch via trunked line
   
2. netgear switch connects to FreeNAS box using 2 trunked lines and a normal ethernet line for admin purposes
3. the FreeNAS box has 7 NICS - 4 are setup as lagg0, 2 are setup as lagg1 (vlan10 / 10.0.2.x passes over this) and 1 is the embedded motherboard NIC which is used for FreeNAS admin (vlan1 / 10.0.1.x)
4. both my laggs are lacp
5. vlan1 is used for administration of all network devices - vlan10 is for my personal, family traffic: plexmedia, owncloud, etc
6. I am looking to pass vlan10 traffic to my two jails over the lagg vlan10 comes in on. vlan1 should not pass any vlan10 data.
7. should I be using VIMAGE and if so, how do I bind it to my vlan interface in FreeNAS?

With regards to point 7: using VIMAGE (from the docs) it looks like enabling it would allow you to configure several different settings, including default gateway for the jails, which is one of the things you need to do here. So yes, I would recommend trying with VIMAGE enabled.

http://www.freebsdonline.com/content/view/742/524/

setfib

this would work perfectly ... has anyone tried this with FreeNAS or does FreeNAS use something similar? I'm now considering maybe I should wipe this machine clean and do a regular setup of FreeBSD 9.3, building a custom kernel

Never personally tried setfib before but this is exactly what I have been trying to say until now lol. I seemed to remember that using defaultroute would be enough, but if you want to try this other way I suppose it won't hurt.

 

[master-richie]

from user kpa in the freebsd.org forums; "Jails can not change the routing table so the setting of defaultrouter in their rc.conf(5) is ignored." - so I cannot use defaultroute

the vlans are correctly configured as interfaces coming over the trunk and when I ping their gateway of 10.0.2.1, I get a response. however unless I change the global default route to 10.0.2.1 I cannot reach the internet. this is also true if I set a static route for that subnet. and of course once I do that my FreeNAS install (10.0.1.10) loses internet.

the problem with vimage, is when using the gui at least I am not able to bind it to a interface.

What you need is a way to tell your router how to route a packet coming from the 10.0.2.0 network to the 10.0.1.1 gateway to access the internet. If the router is configured with an IP address on both networks, you would get this for free.

10.0.1.1 is a virtual device on the router on its own subnet - vlan1. This is what I assumed you understood. vlan10, 10.0.2.1 is another virtual interface on the router. the two NEVER converge.

 

[Deleted47050]

Oh, I didn't know about the defaultrouter setting for jails, bummer. I guess that you must use setfib after all.

Have you tried creating a test VM, put it on the 10.0.2.0 network, assign it to the vlan, configure its default gateway to 10.0.2.1 and see if you can actually reach the Internet? This would help you rule out routing issues as a first troubleshooting step.

 

[master-richie]

I have a HP g5 server that I installed FreeBSD 9.3 on and figured out a clean, native solution to this problem from a pure FreeBSD perspective - even made a tutorial about it: https://forums.freebsd.org/threads/...-route-tables-for-each-jail-setfib-fib.53944/

Due to the ease and success of what I did on that server, I think I'm going to revert my FreeNAS box to FreeBSD 9.3

The short answer is you have to tell your *BSD box in loader.conf to setup more than 1 route table and that route tables should not be shared across FIBs, then in rc.conf assign the extra static routes / gateways to the FIBs that are associated with their respective jails. in each jails.conf file you have to assign ip info and declare which FIB it should use. I do not believe FreeNAS has the equiv in its gui and I don't care to experiment directly with its file system

 

[Deleted47050]

Glad you figured that out :)