Strange network behaviour and LAGG question

[HeMaN68]

New here and have been playing with freenas for a month now. Basic setup, sharing and jails are all working so was now busy to separate the management (WebGUI) and data (smb shares) on the network level with different subnets, and implemented LACP for the data connection.

My Setup:
Dell R510 8 bay / 1* E5620 / 64Gb / Perc H200 IT mode
4 nics (bce0-3)
FreeNAS-11.3-U2.1

my current working network setup is:

bce0-2 aggregated in a LACP connection: lagg2
dhcp enabled
connected to the data network, getting a correct IP from the dhcp server, assume 192.168.1.50

bce3
static ip in the MGMT vlan (vlan set on the switchport), asume 192.168.100.20

bridge0
no ip assigned
bridge members: lagg2 and vnet0.1 (for the jail)
(jail is pulling an ip via dhcp from the data subnet fine like this)

other:
WebGUI IPv4 Adress set to 192.168.100.20 (from dropdown list)
SSH bind interface set to bce3 (from dropdown list)


Issue i have:
1. Unable to select the lagg IP for SMB
In Services - SMB - Configuration I am not able to choose the IP of the LAGG for Bind IP Addresses, only option I have is the IP address 192.168.100.20 (of bce3) which is the adress on the management subnet and not the data subnet.
If I leave Bind IP Addresses blanc smb is also working on the lagg2 IP (192.168.1.50). => I am however not able to bind it to only this IP.

2. network stays down when no ip set on lagg
I tried to solve issue 1 by adding a vlan1 on the lagg2, and also tried setting an IP adress to bridge0.
however, as soon as I remove the dhcp option in the GUI from lagg2 and confirm this setting, all network stays unreachable, even bce3 with the static assigned IP will not come up. It does not matter if I have only the bce interfaces and the lagg created, or the bridge(s) and/or vlan as well. (Thank you devs for the automatic rollback of network changes if not confirmed after committing :) )
As soon as dhcp is removed from the lagg, the network does not come up anymore => I am not able to remove the IP from the lagg so I can add another interface on top of it (bridge or vlan) so I can assign that interface an IP address in the data subnet (and bind that IP to the smb service)


Question:
Is my approach to separating the data and mgmt subnets the right approach, did I miss something or are there other ways to accomplish this?


Thanks for any reply. Always eager to learn :)

 

[Samuel Tai]

Overall, your scheme looks fine. According to /usr/local/lib/python3.7/site-packages/middlewared/plugins/smb.py, lines 134-145,

Code:

    @accepts()
    async def bindip_choices(self):
        """
        List of valid choices for IP addresses to which to bind the SMB service.
        Addresses assigned by DHCP are excluded from the results.
        """
        choices = {}
        for i in await self.middleware.call('interface.query'):
            for alias in i['aliases']:
                choices[alias['address']] = alias['address']
        return choices

the SMB service only allows binding to statically-assigned IPs. You'll have to change lagg2 from DHCP to static addressing.

 

[Samuel Tai]

Also, remember to set a default static route for lagg2 when you transition to static addressing.

 

[HeMaN68]

Thank you, adding a static IP to the lagg was the solution to be able to bind smb to that IP/interface.

What I want to accomplish is that on the data subnet the only freenas-traffic is from my shares.
Things like the management GUI and system updates should be done via mgmt subnet.

I now have set up up Network - Global configuration - IPv4 Default Gateway and Nameserver1 to the data subnet (192.168.1.1)

I think I need to change the Network - Global configuration to use the gateway/nameserver from the mgmt subnet and add a static route for the data subnet.
or should I leave default gateway blanc and in Network - Static Routes add a route for each subnet?
or can I leave out the static route for the data network all together, since it will only be traffic for that subnet and not to leave that subnet at allo no gateway is needed?

So like this:

Global configuration
ipv4 default gateway 192.168.100.1
nameserver1 192.168.100.1

Static routes:
Destination 192.168.1.0 /24 gateway 192.168.1.1 (for data subnet)

or

Global configuration
ipv4 default gateway blanc
nameserver1 192.168.100.1

Static routes:
Destination 192.168.1.0 /24 gateway 192.168.1.1 (for data subnet)
Destination 192.168.100.0 /24 gateway 192.168.100.1 (for mgmt subnet)
Destination 0.0.0.0/0 gateway 192.168.100.1 (for all other traffic, via mgmt gateway. No idea if this will work because this will include both previous routes as well of course)

or

Global configuration
ipv4 default gateway 192.168.100.1
nameserver1 192.168.100.1

Static routes:
none

 

[Samuel Tai]

Without knowing the topology of your network, I can't say which would work better. Typically, however, you should use the default gateway for all your client-side traffic. For the management traffic, you add static routes to your individual management hosts. Don't set defaults on your management interface, as that will cause

1. Interleaving of packets between the client and management VLANs
2. Potentially poor access to the shares, if your management VLAN is locked down.

Depending on how locked down your management VLAN is, you may want to set up a bastion host on the management VLAN, so a static route wouldn't be necessary. Then just connect to the bastion host for all management of the FreeNAS install.

 

[HeMaN68]

thank you for your insights!

network is a home network and two lab servers, pfsense for router/firewall, couple of managed switches and ubiquiti unifi for APs with multiple ssid's.

Because of the lockdown and I started with installing freenas next to my out-of-the box NAS as a project to pass my time. Next step is to secure the network with separare subnets/vlans.
It is all configured at the firewall and switches, but for now security is very open at the firewall.

Next step will be to configure the other servers with separate management and data subnets, and after that tighten the security up.
Also must do something with a separate IOT network :)

I will use the knowledge you provided and see where I will end up