Dedicated interface/IP address for management traffic

[wowbaggerHU]

Hello All!

I'm currently using FreeNAS-9.1.1 on a HP ProLiant Microserver N40L with four (Broadcom) GigabitEthernet interfaces (bge0-1 and bce0-1).

I have the following network setup at the moment:

What I want to achieve is that I want to separate management-related traffic to the interface belonging to the management VLAN (bce0), and the NAS traffic to the generic VLAN (bge1).
I have a host in the generic VLAN (hostA), from which I want to be able to manage the NAS, and also be able to use its NAS services.

Currently however, accessing the NAS via HTTP/HTTPS on 172.18.88.171 doesn't work from hostA. (hostA's default gateway is 172.18.88.1 and freenas' is 172.18.88.161.)
Let me describe the problem briefly:
- hostA sends the SYN packet through the router, to bce0
- freenas answers with SYN-ACK on bge1
- hostA sends the ACK through the router (just as before)
- the router's (SPI) firewall drops the ACK, because it hasn't seen the SYN-ACK that was sent by freenas directly to hostA via bce0.

I've discussed this problem with some colleagues more knowledgeable than me, and they told me to try policy-based routing. In the meantime I also found one other possible solution for this, the use of multiple routing tables.
Unfortunately none of these two approaches do work currently with FreeNAS, because the multiple routing tables feature would require the recompilation of the kernel with the appropriate option enabled. (The default kernel contains support for only one routing table.)
Also, for the policy based routing to work, I would need to rely on pf, which isn't included in FreeNAS either.

What possible solution do you FreeNAS people recommend for my problem?
Thanks in advance for your answer!

Note 1: If I add a host route pointing to 172.18.88.6 for 172.18.88.171 on hostA, the web interface works. (This is a dirty solution, I don't really like it.)
Note 2: If I try to add a host route pointing to 172.18.88.6 for 172.18.88.171 on the router, and a host route pointing to 172.18.88.1 for 172.18.88.50 on the FreeNAS box, then it doesn't work either, because FreeNAS already has an automatically generated routing entry for 172.18.88.50, which I am unable to remove. (If I could force FreeNAS to communicate to 172.18.88.50 via the router, then this would be a bit better than the solution presented in Note 1, but it still isn't a good one IMO.)

 

[cyberjock]

That spider profile picture reminds of the hilarious story behind it.

Are you sure that those subnets aren't overlapping? I didn't do the math, but I think those 2 VLANs are conflicting with each other. Am I wrong?

 

[wowbaggerHU]

That spider profile picture reminds of the hilarious story behind it.

Are you sure that those subnets aren't overlapping? I didn't do the math, but I think those 2 VLANs are conflicting with each other. Am I wrong?

Yes, I am positive.
172.18.88.0/25 goes from 172.18.88.0 to 172.18.88.127, while 172.18.88.160/27 goes from 172.18.88.160 to 172.18.88.191.

 

[cyberjock]

Yeah, you are right. I just checked your math to make sure. My guess is the routing needs to be changed. But to be honest, I have no clue how you'd actually change the routing.

If you do netstat -nr you'll see the current routing. In my case, the routing is pretty much "go to NIC1 for everything".

Can you post the output of netstat -nr?

As a sidenote, even if we can fix this with a routing table, I don't know how we are going to fix it in a permanent way. FreeNAS doesn't have the ability built-in to do what you want to do, I don't think.

Did you try setting up VLANs? I'm very unfamiliar with VLANs and how it actually works in FreeNAS. Might be something you want to try. I don't have 2 networks myself so I can't really experiment with VLANs very well.

If you do get it working please provide some screenshots of what settings you used!

 

[wowbaggerHU]

Yes, sure. My routing table looks like this:

[root@space ~]# netstat -rn -f inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 172.18.88.161 UGS 0 82 bge1
127.0.0.1 link#12 UH 0 3296 lo0
172.18.88.0/25 link#1 U 0 1295 bce0
172.18.88.6 link#1 UHS 0 0 lo0
172.18.88.160/27 link#4 U 0 28 bge1
172.18.88.171 link#4 UHS 0 0 lo0
172.18.90.0/28 link#13 U 0 0 lagg0
172.18.90.1 link#13 UHS 0 0 lo0
[root@space ~]#

Well, I think that just compiling the kernel with the multiple routing tables feature enabled*, or compiling and shipping pf.ko in the next release would make it possible to make it work for me.**

*: This wouldn't break anything, because by default everything uses rouing table 0, so this change would be transparent to users and processes.
**: Fixing my problem would involve some custom scripting, because IMO some/most of the settings that need tinkering can't be modified from the web interface. Thus my goal would be to create a self-contained script, that would carry out all the neccessary changes to the config, and even restarting some of the services, or adding the pf rules required for policy-based routing.

I'm currently using VLANs, though only on the switch and the router. Adding VLAN tagging to the interfaces on the FreeNAS box wouldn't really change the big picture here, because this routing problem would still be present.

 

[cyberjock]

To be honest, its not recommended you start doing things like custom scripts. Things get really messy and it could potentially break alot of things. To be honest you might be better off with full fledged FreeBSD if you want/need to go that route. As a general rule we recommend against people using the CLI unless they have to, and even then be very cautious about what you choose to do. The FreeNAS config does alot of stuff automatically and there may be unforseen problems long term from doing what you are proposing.

I know this isn't much help right now, but you might want to put a ticket in at bugs.freenas.org and request that the multiple routing tables function be added to FreeNAS.

 

[wowbaggerHU]

Okay, I understand why you are warning me.

BTW, do you know, or can you guess which feature is more likely to be implemented if I request it; the one involving the addition of pf, or the one enabling more routing tables?

 

[Dusan]

While pf isn't included, ipfw is (it is used by warden/jails). It seems it is also able to do policy-based routing. Maybe ipfw will allow you to accomplish what you need:
https://www.freebsd.org/doc/handbook/firewalls-ipfw.html
https://www.freebsd.org/cgi/man.cgi?query=ipfw

 

[cyberjock]

Okay, I understand why you are warning me.

BTW, do you know, or can you guess which feature is more likely to be implemented if I request it; the one involving the addition of pf, or the one enabling more routing tables?

That's a tough one. It's really hard to say as it depends on what iXsystems deems a priority. They are the retail branch of FreeNAS and sell it as TrueNAS. Your best bet if you are a programmer is to do a pull request with FreeNAS on github after you get it working for yourself. pf.ko should be relatively easy to pull into FreeNAS if you have the source. And all you need is for it to be compiled with FreeNAS and you could use it yourself. 9.2.0 is scheduled to go to RELEASE soon so if you can get pf.ko compiled with FreeNAS you could at least use it yourself. But, considering how late in the development cycle it is unless you plan to do the pull request yourself it's like to be ignored until after 9.2.0 is released. :(

 

[Dusan]

Your best bet if you are a programmer is to do a pull request with FreeNAS on github after you get it working for yourself. pf.ko should be relatively easy to pull into FreeNAS if you have the source.

You don't need to search for the sources as pf is included in the FreeBSD source tree. It is a one line change to get it compiled. However, as I already mentioned above, this may not be necessary. pf is the OpenBSD packet filter (that was ported to and is included in FreeBSD), however the "native" FreeBSD packet filer ipfw should be perfectly capable doing the same. ipwf is included in FreeNAS.

 

[cyberjock]

Dayum! Dusan is on fire today! Busting out the moves!

 

[wowbaggerHU]

You don't need to search for the sources as pf is included in the FreeBSD source tree. It is a one line change to get it compiled. However, as I already mentioned above, this may not be necessary. pf is the OpenBSD packet filter (that was ported to and is included in FreeBSD), however the "native" FreeBSD packet filer ipfw should be perfectly capable doing the same. ipwf is included in FreeNAS.

Thanks for the info. I seem to have overlooked that ipfw is enabled by default.
Looking at the output of kldstat -v, I found that the ipfw and ipfw_nat modules are complied into the kernel.
Some of the tutorials mention, that I need the IPFIREWALL_FORWARD option enabled in addition to IPFIREWALL.
How can I check whether the former is enabled in the kernel?

I tried this:
[root@space] ~# sysctl net.pfil.forward
sysctl: unknown oid 'net.pfil.forward'

 

[Dusan]

How can I check whether the former is enabled in the kernel?

Check the source code :). It is enabled:
https://github.com/freenas/freenas/blob/master/nanobsd/FREENAS.amd64#L86

 

[wowbaggerHU]

Check the source code :). It is enabled:
https://github.com/freenas/freenas/blob/master/nanobsd/FREENAS.amd64#L86

Cool! Thanks for the hint!

 

[Dusan]

Btw, this may also help: https://wiki.freebsd.org/IpfwNg#What_is_the_problem_with_dynamic_rules.3F
It shows how to implement the the pf reply-to behavior in ipfw.