Many FIBs, issue SMB on the same network

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
Hi,

I'm sorry if, I'm in the wrong category.

I've just set up many fib (3). I'm going to explain my configuration.
1 pfsense (3vlans, 101, 102, 103) > Freenas (lacp, vlan 101, 102, 103)

I set up :
vlan101(mgmt) fib0 : 10.1.0.0/24, bound GUI, ssh, defaut gateway 10.1.0.254
vlan102(user) fib1 : 10.2.0.0/24, bound SMB, defaut gateway 10.2.0.254
vlan103(DMZ) fib2 : 10.3.0.0/24, bound jails, defaut gateway 10.3.0.254

My issue, is when I used smb. If, I am in other network 10.1 or 10.3, that works.
But whan I am in 10.2 I received non SYN, ACK.

root@jokul:~ # setfib 0 netstat -rn
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 10.1.0.254 UGS vlan101
10.1.0.0/24 link#5 U vlan101
10.1.0.3 link#5 UHS lo0
127.0.0.1 link#3 UH lo0


root@jokul:~ # setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination Gateway Flags Netif Expire
default 10.2.0.254 UGS vlan102
10.2.0.0/24 link#6 U vlan102
10.2.0.1 link#6 UHS lo0
127.0.0.1 lo0 UHS lo0

root@jokul:~ # setfib 2 netstat -rn
Routing tables (fib: 2)

Internet:
Destination Gateway Flags Netif Expire
default 10.3.0.254 UGS vlan103
10.3.0.0/24 link#7 U vlan103
10.3.0.1 link#7 UHS lo0
127.0.0.1 lo0 UHS lo0


root@jokul:~ # sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS

root smbd 8620 32 tcp4 10.2.0.1:445 *:*
root smbd 8620 33 tcp4 10.2.0.1:139 *:*
root smbd 8620 34 tcp4 127.0.0.1:445 *:*
root smbd 8620 35 tcp4 127.0.0.1:139 *:*
root nmbd 8616 16 udp4 *:137 *:*
root nmbd 8616 17 udp4 *:138 *:*
root nmbd 8616 18 udp4 10.2.0.1:137 *:*
root nmbd 8616 19 udp4 10.2.0.255:137 *:*
root nmbd 8616 20 udp4 10.2.0.1:138 *:*
root nmbd 8616 21 udp4 10.2.0.255:138 *:*
root uwsgi-3.6 3952 3 tcp4 127.0.0.1:9042 *:*
root uwsgi-3.6 3609 3 tcp4 127.0.0.1:9042 *:*
www nginx 3539 6 tcp4 10.1.0.3:4443 *:*
root nginx 3538 6 tcp4 10.1.0.3:4443 *:*
root ntpd 3193 21 udp4 *:123 *:*
root ntpd 3193 24 udp4 127.0.0.1:123 *:*
root ntpd 3193 25 udp4 10.1.0.3:123 *:*
root ntpd 3193 26 udp4 10.2.0.1:123 *:*
root ntpd 3193 27 udp4 10.3.0.1:123 *:*
root sshd 3113 4 tcp4 10.1.0.3:4222 *:*
root sshd 3113 5 tcp4 127.0.0.1:4222 *:*


Server to client :

root@jokul:~ # setfib 1 ping 10.2.0.9
PING 10.2.0.9 (10.2.0.9): 56 data bytes
64 bytes from 10.2.0.9: icmp_seq=0 ttl=128 time=0.348 ms
64 bytes from 10.2.0.9: icmp_seq=1 ttl=128 time=3.178 ms
64 bytes from 10.2.0.9: icmp_seq=2 ttl=128 time=0.563 ms
64 bytes from 10.2.0.9: icmp_seq=3 ttl=128 time=0.371 ms


In attachment my wireshark

I'm totally blind about this trouble
 

Attachments

  • bad.PNG
    bad.PNG
    68.6 KB · Views: 296
  • good.PNG
    good.PNG
    95.4 KB · Views: 293
Last edited:

proto

Patron
Joined
Sep 28, 2015
Messages
269
what happens when you disable lacp?
is there a switch between your pfSense and FreeNAS?
 

foxi352

Cadet
Joined
Apr 21, 2019
Messages
8
Hi @andfrid,

Sadly i have no response for you but questions. I am new to FreeNAS and relatively new to FreeBSD but coming from a linux world.
I have a very similar setup than yours. PfSence, 5 VLAN's and 4-ports LCAP to FreeNAS.

But i am confused about fibs. I know it is to have separate routing tables, but that's all. Can you quickly tell me how you did your setup ?

What i did so far:
  • Created all VLAN's in GUI (100, 200, 300, 500, 999)
  • Created "persistent" bridges in tunables like for example: ifconfig_bridge500 / addm vlan500 up
  • Created plugins / jails and set (example) interface to vlan0:bridge500 and vnet to vlan500 and allowed raw_sockets for icmp.
GUI is on VLAN 100 and default gateway is also set to GW on VLAN 100.
I can access my GUI and all jails on VLAN 100 from my workstation (also in VLAN 100)

All other jails on VLAN 500 for example are up and running. bridge500 is there. vnet0:whatever and vlan500 are members. But jail is not accessible from outside and from inside jail i am isolated. Routing table is there with correct default gateway for vlan500 but i can't even ping the GW IP from inside jail: Host is down.

So i feel that i am very close, but i may miss correct setup for these FIB things. Where did you set this up ?

I tried to add the following to tunables, but this did not change anything:
  • net.add_addr_allfibs to 0 in sysctl
  • net.fibs to 10 in loader
Maybe you could point me to the right direction for this last step ?
 

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
what happens when you disable lacp?
is there a switch between your pfSense and FreeNAS?

Why should I disable LACP, it works fine. yes, there is a L2 switch between pfsense and freenas.

foxi352, I will ask you to night (for me, in Europe).
 

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
So,
I don't know if it's the best method because, it doesn't work like I want with SMB, for example.

step 1 in Tunables GUI :
net.add_addr_allfibs 0 sysctl >>This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.
net.fibs 3 loader >> number of FIB do you want

Step 2 still in Tunables GUI (Set up your routes) :
static_routes lan1 lan2 lan3 rc.conf
route_lan1 default @ipgwlan1 -fib 1 rc.conf
route_lan1 default @ipgwlan1 -fib 1 rc.conf

Step 3 : In network > Interface
Add in interface's option fib 1 on the good interface

Step 4 : restart

Step 5 : Configure your jail :
Vnet0, IP/submask/Gateway
Leave vnet0:bridge0 and select default interface on the good vlan in Jail's network's properties

Step 6 : enjoy
It's works for me.
 

foxi352

Cadet
Joined
Apr 21, 2019
Messages
8
Many thanks andfrid. Everything works now, but i had some problems which i want to share with you. Maybe it helps you or another reader of this thread.

I had to use a different bridge per VLAN to get it working correctly:

For example for VLAN200 and VLAN500 i did:

Tunables:
static_routes vlan200 vlan500 rc.conf
route_vlan200 default 192.168.2.254 -fib 2 rc.conf
route_vlan500 default 192.168.5.254 -fib 5 rc.conf

Network->Interfaces:
VLAN200: fib2
VLAN500: fib5

In jail depending in what vlan you want it to be:
interfaces: vnet0:bridge200 or vnet0:bridge500
vnet_default_interface: vlan200 or vlan500

I have my Web GUI in VLAN100. For that VLAN to work i had to remove fib and everything in tunables. Just had to set the IP address in Network->Interfaces for the entire VLAN100 to work. It is on fib 0 and bridge 0.

Everytime i changed something i had to reboot to get it to work correctly.
 

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
Hi,
In my side, I don't set fib 0 on my GUI interface because it's default fib.


In jail, I have different jails in different networks but I leave vnet0:bridge0 for all my jails
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
Why should I disable LACP, it works fine. yes, there is a L2 switch between pfsense and freenas.
You should not disable LACP if it's working.
Your net configuration is similar to mine, then, apart that particular fib setup and SMB issue on the same subnet... Does pfSense report anything strange?
 

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
Hi proto

No, I see nothing in pfsense.
How is your freenas network set up ? Don't you set up many FIB with many vlans ?
 

foxi352

Cadet
Joined
Apr 21, 2019
Messages
8
Hi,
In my side, I don't set fib 0 on my GUI interface because it's default fib.
Yes, that's what i meant and did too. Sorry if i was not clear on this one.
In jail, I have different jails in different networks but I leave vnet0:bridge0 for all my jails
Even if that works, i am not so sure about the security point of view having everything on the same bridge. But then, i am not the FreeBSD networking specialist, so maybe i am completely wrong. But i feel more secure having separate bridges, even if it's purely psychological :smile:
 

andfrid

Dabbler
Joined
Apr 18, 2018
Messages
25
Yes, that's what i meant and did too. Sorry if i was not clear on this one.

Even if that works, i am not so sure about the security point of view having everything on the same bridge. But then, i am not the FreeBSD networking specialist, so maybe i am completely wrong. But i feel more secure having separate bridges, even if it's purely psychological :)

I'am not too and I don't know how is working it with different bridges. I would like to know what is the best way to do.
 

proto

Patron
Joined
Sep 28, 2015
Messages
269
No, I see nothing in pfsense.
How is your freenas network set up ? Don't you set up many FIB with many vlans ?

No, I don't use fibs. For me it's still something esoteric : - )
Anyway I see traffic flowing from routes on jails/vms, via bridges.

What I'm really unable to achieve is having a persistent bridge configuration like @foxi352. When I configure those tunables my network configuration breaks: no VLANs, no networking @ reboot... So I have to survive with bridge configuration generated by middleware/netif during jails/vms configuration and deal with them.

But we should first check our FreeNAS version / NICs:
- FreeNAS 11.2-u5
- igb | em

Now the long and boring part : - ) It's a long story of trials and errors and (permanent) failures.

This is my network setup actually. Take it AS IS, I'm in a testing loop and probably it will not survive this hot summer!
I really don't need LACP, so I configured a double LACP just for fun.
I didn't configured IP VLANs, except for VLAN10.

Code:
# current setup as of 2019 07 29

Router  -> pfSense -> hp v1920 48 ports -> FreeNAS

VLANs:
- 10 MNG net
- 20 Client
- 100 PROD / TODO
- Other VLANs

pfSense: 4 Nics + VLANs
igb0 --> WAN
igb1 --> LAN20 / Default LAN IF
igb2 --> PROD / Actually TODO
igb3 --> LAN10
     --> Other VLANS

Switch:
2 Dynamic LACP:
1) trunk ports with TAGGED VLANs 10 / 20
2) trunk ports with TAGGED VLANs 100 / Other

FreeNAS: 4 Nics (2 mb igbX + dual nic emx) + IPMI
1) lagg0 - LACP
   - igb0 | igb1
2) lagg1 - LACP
   - em0 | em1
Note: I think they should be "crossed" but I have no other igbX NICs.
I mean: one mb NIC should be in LAG with a PCIe NIC!

FreeNAS admin interface is bound to VLAN 10:
- static IP / Default route only to pfSense interface / DNS pointing to internal DNS (pfSense again)
I have a mixture of Jails and Bhyve VMs running on different VLANs.

Ex:
bridge0:
    member: tap5
    member: vnet0:4
    member: vnet0:3
    member: vnet0:2
    member: tap3
    member: tap2
    member: tap1
    member: vlan10
    member: tap0

bridge1:
    member: tap4
    member: vnet0:1
    member: vlan20

bridge2:
    member: vlan25
Note: actually VM id down.


Jails configuration:
Code:
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+
| JID |  NAME  | BOOT | STATE |   TYPE   |     RELEASE     |         IP4         | IP6 | TEMPLATE |
+=====+========+======+=======+==========+=================+=====================+=====+==========+
| -   | clamav | off  | down  | pluginv2 | 11.2-RELEASE-p9 | DHCP (not running)  | -   | -        |
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+
| 4   | edms   | off  | up    | jail     | 11.2-RELEASE-p9 | epair0b|10.10.10.17 | -   | -        |
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+
| 2   | minio  | off  | up    | jail     | 11.2-RELEASE-p9 | epair0b|10.10.10.16 | -   | -        |
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+
| 3   | odoo   | off  | up    | jail     | 11.2-RELEASE-p9 | epair0b|10.10.10.15 | -   | -        |
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+
| 1   | test20 | off  | up    | jail     | 11.2-RELEASE-p9 | epair0b|10.10.20.61 | -   | -        |
+-----+--------+------+-------+----------+-----------------+---------------------+-----+----------+


So: I have JID 1 (test20) in VLAN20 --> bridge1
vnet0:1 is epair0b
JID 2-4 in VLAN10 --> bridge0

test20 JID1 jail:
Code:
interfaces:vnet0:bridge1
ip4:new
ip4_addr:none
ip4_saddrsel:1
vnet_default_interface:auto
vnet_interfaces:none


Other jails 2-4:
Code:
interfaces:vnet0:bridge0
ip4:new
ip4_addr:none
ip4_saddrsel:1
vnet_default_interface:auto
vnet_interfaces:none


Routing:
Code:
# JID 1 - test20
iocage exec test20 "netstat -rn"
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.10.20.1         UGS     epair0b
10.10.20.0/26      link#2             U       epair0b
10.10.20.61        link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0

# JID 2- minio
iocage exec minio "netstat -rn"
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.10.10.1         UGS     epair0b
10.10.10.0/26      link#2             U       epair0b
10.10.10.16        link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0


ALL Jails are getting DHCP from pfSense and bpf:yes | vnet:on | dhcp:on

Same for VMs: their taps all get address from DHCP and they works out of the box once the correct VLAN is specified in Device panel.

But I must admit some issues too on VLAN20/Client jails. If I create a bridge manually those jails in VLAN20 don't get address because they do not receive a DHCPACK. I should investigate, but it's not a real issue as I will remove that Client VLAN20 from the NAS.

Now. A simple test: Disconnect/Disable LAG ports from switch. What happens? If i disconnect LACP-1 (VLAN10/VLAN20) networking is interrupted, but VLAN100 and Others are still working as they are on LACP-2. What's the meaning? Their routes are perfectly working even if I don't set "physical" FIB routing.
Just check with systat -ifstat to see traffic flowing from nics/bridges.

Now... Services. NAS serves only on VLAN10, so other nets have to pass firewall rules.
SMB is accessible via VLAN10 only and port 445 only, same for SSH and GUI. On pfSense I have a simple floating rule: SMB is allowed from LAN20 only.

LACP considerations.
If it's not well configured (but I lack a lot of networking knowledge to confirm): it can disrupt networking in a bad manner. Think of a network GUI refresh or SSH packet loss... This could be a symptom of misconfiguration.
If there are some clients it's not useful. LACP is good when you have a lot of concurrent connections. Most of the time I don't have many clients connected and I'm thinking of rolling back to single links configuration.
 
Top