pfSense + two LAN subnets + FreeNAS shares

[alobi]

Hi guys . I'm having issues with pfSense, FreeNAS, and two NICs - I'm hoping someone has some sage advice for me.

I have a network setup that looks like this:

Code:

                 NIC A                   
                       --> switch --> FreeNAS
                      | 
                      |                  
internet -> pfSense --                   
                      |				 --> FreeNAS
                      |				|
                       --> switch --
                 NIC B     			|
                       				 --> iMac

My iMac is part of LAN subnet 192.168.2.x, and FreeNAS (with two separate NICs) is a member of both subnets 192.168.1.x and 192.168.2.x. I have two rules at the top of my pfSense firewall, one to allow all traffic from LAN A to LAN B and the other to allow all traffic from LAN B to LAN A. This all works fine, and I can SSH, ping, access shares, etc. across subnets. The reason FreeNAS has two NICs is to assign certain jails to LAN A (a VPN connection) and others to LAN B (clear net).

My problem is subtle but frustrating. As I said, my devices can talk to FreeNAS across subnets, but the connections are fragile. My SSH connection will drop with a broken pipe after ~70 seconds, even if the connection isn't idle. My AFP shares disconnect frequently, causing lots of BBoDs (beach balls of death) on the Mac accessing the shares while it tries to restore the connection. Even accessing FreeNAS from the web GUI has issues - my Mac on LAN B attempts to connect to FreeNAS on LAN A via web GUI but the browser can never load the entire page. It ends up looking like the attached image.

Accessing FreeNAS from the same subnet works fine though. SSH connections don't drop, the web GUI loads, etc.

I've isolated this to FreeNAS to the best of my knowledge - my reasoning being I can SSH to another Mac on a different subnet (the same subnet that FreeNAS lives on), without any broken pipes.

I've been racking my brain on this for a while and I'm stuck. Any ideas?

 

[anodos]

Have you assigned static addresses to the NICs or are you using dhcp?

Personally, I'd put both of your FreeNAS server on LAN B only, and set up routing / filtering rules for traffic from LAN A to LAN B. There's no need to put it physically on both networks.

 

[Tywin]

Accessing FreeNAS from the same subnet works fine though. SSH connections don't drop, the web GUI loads, etc.

I've isolated this to FreeNAS to the best of my knowledge - my reasoning being I can SSH to another Mac on a different subnet (the same subnet that FreeNAS lives on), without any broken pipes.

The one thing you haven't isolated here, and that you aren't really able to while keeping the same topology, is pfSense itself. Since this problem only occurs on connections being routed through pfSense, I would start there.

Do you have any other servers that you can try SSHing to across LANs? How large is your network? My gut reaction is that you are making your life more complicated that it needs to be.

 

[alobi]

Thanks for the replies!

Have you assigned static addresses to the NICs or are you using dhcp?

Personally, I'd put both NICs on LAN B in LACP (though you probably won't benefit from it) and set up routing / filtering rules for traffic from LAN A to LAN B. There's no need to put a NIC in both.

I have pfSense acting as a DHCP server for both LAN subnets so FreeNAS has two static IPs (one for each NIC) and the iMac has one as well. The reason I used separate NICs was to be assured that I was isolating the traffic for some of my jails. That's really my one and only goal for this entire venture, to route 100% of the traffic in and out of certain jails over the LAN that I specify.

The one thing you haven't isolated here, and that you aren't really able to while keeping the same topology, is pfSense itself. Since this problem only occurs on connections being routed through pfSense, I would start there.

Do you have any other servers that you can try SSHing to across LANs? How large is your network? My gut reaction is that you are making your life more complicated that it needs to be.

I am able to SSH to my Macbook across subnets without any broken connections (which is also routed through pfSense), which is why I thought it could be a FreeNAS-related thing. However, I just tried disabling my pfSense firewall rules completely and the SSH connection to my FreeNAS machine across subnets doesn't break, which is unexpected and confusing. Is there any way I can still blame it on FreeNAS, given the previous Macbook SSH example? My firewall rules are not (to my knowledge) convoluted. I'll attach screenshots of what they look like.

Maybe I am making things complicated. I'm all for simplicity! But this is the only way I could think of to route all the traffic of some jails over a specific LAN (VPN) connection. And using a 2nd physical NIC felt safer. Is there a better way?

 

[pirateghost]

My guess is that your dhcp lease is also handing out default gateways. A computer can only have one gateway. If your pfsense box is handing out a gateway for each nic, you are confusing FreeNAS as to which gateway is real. Set static IP addresses on FreeNAS and only apply one gateway to one nic. See what happens.

 

[alobi]

My guess is that your dhcp lease is also handing out default gateways. A computer can only have one gateway. If your pfsense box is handing out a gateway for each nic, you are confusing FreeNAS as to which gateway is real. Set static IP addresses on FreeNAS and only apply one gateway to one nic. See what happens.

Thanks for the info. That makes perfect sense to me. I gave it a shot:

- removed the entries for each FreeNAS NIC in the pfSense DHCP Server settings
- made sure the default gateway being handed to the clients is that of the router itself (192.168.1.1)
- assigned static IPs in the Network -> Interfaces area of FreeNAS (screenshot attached)
- assigned FreeNAS a default gateway of 192.168.1.1 in Network -> Global Configuration

It doesn't look like that worked :-/ SSH connection still drops and the web GUI is still half-broken when accessing it from the opposite subnet.

 

[alobi]

Previous posts got me wondering about my networking setup on the FreeNAS side. I'm not a routing wiz, but do these routes look correct?

Code:

➜  ~  netstat -rn                                
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0        0   igb0
127.0.0.1          link#6             UH          0      155    lo0
192.168.1.0/24     link#3             U           0       41   igb0
192.168.1.20       link#3             UHS         0        0    lo0
192.168.2.0/24     link#1             U           0      101    em0
192.168.2.20       link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#6                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%lo0/32                     ::1                           U           lo0

 

[alobi]

OK I think I am getting close. I've solved the gateway mismatch/broken pipe issues by removing the static IP from the LAN B (henceforth igb0) NIC, leaving only the LAN A (henceforth em0) NIC with a set IP (which is the IP I access the webGUI with). But now the jails that are on LAN B are having networking issues.

igb0 can be considered the main physical network interface - it has a static IP and gateway assigned. All new jails by default will use that NIC and gateway, etc. But I have a few jails that I want to segregate onto a separate physical NIC (em0) that's part of a different subnet and needs to route through a different gateway. I can't seem to use the new option in 9.3 that lets you specify a NIC for each jail because it will still inherit the default route from FreeNAS, which is not the one I want to use:

Code:

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.2.1        UGS         0      888   igb0
127.0.0.1          link#6             UH          0     8554    lo0
192.168.1.0/24     link#1             U           0        0    em0
192.168.1.21       link#1             UHS         0       89    lo0
192.168.2.0/24     link#3             U           0     7644   igb0
192.168.2.20       link#3             UHS         0      318    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#6                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%lo0/32                     ::1                           U           lo0

I want the jail's default route to point to 192.168.1.1, but it's picking up the gateway from the FreeNAS box which is 192.168.2.1 (and the default Netif is not the one I assigned in the jail settings - why is that?). And I can't manually edit the routes since VIMAGE is disabled for the jail.

Any suggestions?