Jail network (for UniFi controller) setup

[Aephir]

I created a jail and (based on FreeBSD 13.1) an installed UniFi controller (pkg install unifi7), and I'm pretty sure I'm doing something wrong with the network setup.

I have set up a link aggregation (lagg1) for the TrueNAS system, and the jail is configured like my jail for Nextcloud (set up with the "all-in-one" script):

Basic properties:
None of the boxes checked (I initially checked the VNET, but then I couldn't install packages or ping names, only ping IPs (8.8.8.8 but not google.com)).

Code:

vnet_default_interface: auto
ip4_interface: lagg1
ipv4_address: 10.0.30.68 (my TrueNAS is on 10.0.30.27 on a 10.0.30.0/24 VLAN/subnet)
ipv4_netmask: 24

Network properties:

Code:

interfaces: lagg1:bridge0

Although I have tried many combinations of changing the settings above, with no luck. The closest I got was once, when I navigated to https://10.0.30.68:8843 where it said "starting network application" (or something like that) briefly before it showed "HTTP Status 400 - Bad Request" upon refreshing (I had "Network properties" --> "interfaces" set to "lagg1:bridge0", but all else as shown above).

But other than that, I only see "HTTP Status 400 - Bad Request" if navigating to either http://10.0.30.68:8880 or https://10.0.30.68:8843.

I've searched around a bit (incl. finding this post), but I can't seem to find a setup that works. Any help/suggestions would be appreciated.

(EDIT: the jail gets the correct IP, and I can see it pop up in my network in my old Unifi controller running as a Home Assistant addon)

 

[Patrick M. Hausen]

Did you statically create your bridge0 interface instead of letting TrueNAS auto create it? You should.

Your settings should then read:

Code:

vnet_default_interface: none
ip4_interface: vnet0
ipv4_address: 10.0.30.68
ipv4_netmask: 24


interfaces: vnet0:bridge0

 

[Aephir]

I had not before, no. I just did now, with "default" settings, then added settings as you described (only checking "VNET" as per screenshot in your other post) to a completely new jail.

The jail now has no network, does not show up in Unifi controller, and can't ping either 8.8.8.8 or google.com.

I'm guessing I need more than just the "default" setup (only adding name = "bridge0") when creating the bridge?

EDIT: I just saw your post about having vnet_default_interface set to "auto", and realized that is exactly how the all-in-one script creates the nextcloud jail. But even after changing that from "auto" to "lagg1" and rebooting, there's still no network. for the new jail.

 

[Patrick M. Hausen]

You need to add lagg1 as a member to bridge0 and move all IP address configuration from lagg1 to bridge0 instead. IP addresses MUST be on the bridge interface.

 

[Aephir]

I tried setting up the bridge (incl. lagg1 as member, saved settings after testing, removed "dhcp" from lagg1, and added it to bridge0).

In jail settings, if I use settings as you described above (I can only choose "vnet0" in "basic properties" if I first write "vnet0:bridge0" in "network properties", then save, open settings again and go back to "basic properties"?), there's no network access.

It shows up on network with correct ip, but I can't ping internal or external IPs.

Edit: Same for the Nextcloud jail. It shows up on network, but I can't access it. Just in case I did something wrong, here's the unifi jail settings (I had to add 10.0.30.1 as ipv4 default router for the jail to even start):

 

[Patrick M. Hausen]

Configure bridge0 statically and remove DHCP from lagg1. This is documented in the FreeBSD handbook but nowhere in the TrueNAS documentation for ... reasons. A bridge member interface must not have an IP address in FreeBSD. After any changes in the network settings your jails will lose connectivity and you need to reboot to restore it. But this "dance" is only necessary once.

 

[Aephir]

Ok, so just to make sure, the steps I took were:

• Remove dhcp from lagg1
• Create bridge0, add lagg1 as member and set an IP in settings (e.g., 10.0.30.27/24, my old dhcp reservation). No dhcp enabled.
• Save settings and reboot TrueNAS
• In jail settings, "Basic Properties":
  • Check only "VNET" of the four checkboxes
  • Set the following:

Code:

vnet_default_interface = none
ipv4_interface = vnet0
ipv4_address = 10.0.30.68/24
ipv4_router = 10.0.30.1

• In jail settings, "Network Properties":
  • Set the following:

Code:

interfaces = vnet0:bridge0

• Leave everything else "as was" when jail was created.
• Start jail.

With that, both the TrueNAS and jails show up on network with correct IP, but I can (from within jail) only ping anything else on the same TrueNAS hardware, nothing else on LAN/WAN, and Nextcloud UI is no longer accessible.

 

[Patrick M. Hausen]

Did you disable hardware offloading on the physical member interfaces of yor lagg1? Did you put "up" into the options field of these?

 

[Aephir]

I had not disabled hardware offloading (mostly because of the warning of "severely reducing network performance"; this is not something I should be worried about for the base NAS usage?).

So, in addition to the list above:

• Check "Disable hardware offloading" for both physical interfaces of the lagg1 (bge0 and bge1 in my case)
• Add "up" in "Options" of both bge0 and bge1

• Saved settings and rebooted again.

I created a new jail, just to be sure I hadn't messed something up with all the changes back and forth, but no change. I can ping TrueNAS IPs (10.0.30.27, 10.0.30.67, etc.), but no other IPs from within jail.

 

[Patrick M. Hausen]

And the lagg1 is configured how? And connected to the switch how?

No need to worry about that warning. Performance will not degrade, but your CPU will have a little bit more to do. Yes to "up" and checking that box for both bge0 and bge1.

 

[Aephir]

lagg1 is lcap. Do I need to disable hardware offloading on the lagg1 as well?

What do you mean "connected to the switch how?"

In case it helps troubleshooting, I noticed with the jails are up, showing up on network, they are not listed under IP for the bridge (should they be?):

They were when I used lagg1 without the bridge.

 

[Patrick M. Hausen]

The lagg1 needs to be connected to a switch that has LACP configured on both ports, too.

Please post the output of ifconfig on the NAS host. Thanks.

 

[Aephir]

Code:

root@truenas[~]# ifconfig
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:24:7e:54:6f:72
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        hwaddr 00:24:7e:54:6f:73
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
lagg1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Link aggregation LCAP
        options=80098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:24:7e:54:6f:72
        laggproto lacp lagghash l2,l3,l4
        laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:45:74
        inet 10.0.30.27 netmask 0xffffff00 broadcast 10.0.30.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: lagg1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:24:7e:ba:b5:81
        hwaddr 02:8f:8e:f6:67:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.10.10.1 --> 10.10.10.2 netmask 0xffffff00
        groups: tun
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 2039
vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: Test as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:24:7e:1b:34:d2
        hwaddr 02:c5:89:c4:91:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>

 

[Patrick M. Hausen]

Most of that looks good. The lagg hash needs to match the setting on the switch. Few switches do l4 - most probably l2,l3 is the correct setting, but I would check. As I spot a tun0 - any firewall and/or NAT rules active, possibly?

 

[Aephir]

The switch connecting bge0/bge1 is a UniFi Switch Lite 8 PoE, so L2.

Do I need to change anything on the TrueNAS side to accommodate? I didn't when first setting up, and just using lagg1 (no briidge) has worked pretty well until now.

Good catch!! "tun0" is for my OpenVPN. I have the following "tunables" set up for the OpenVPN server (incl. firewall and nat):

Code:

firewall_enable = yes
firewall_type = open
gateway_enable = yes
ipv6_gateway_enable = yes
natd_enable = yes
natd_flags = -dynamic -m
natd_interface = lagg1

I changed "natd_interface = lagg1" to "natd_interface = bridge0", and there's a bit of progress: Now the jails can at least ping IPs (both 10.0.30.1, but also external like 8.8.8.8).

They still can't ping google.com though (or freebsd.org, and I can't install pkg or packages), and I don't have access to the Nextcloud web UI.

I don't have any additional firewall or NAT settings on the UniFi side, I'm basically running with vanilla rules (just the WAN rules) for now. The Home Assistant addon version of Unifi controller I'm using is very limited compared to my old EdgeRouter X, and I can't make the NAT, masquerade, etc. settings I need (I need an UniFi controller where I can utilize the config.gateway.json file).

That's actually the entire reason for trying to set up a UniFi controller in a jail.

 

[Patrick M. Hausen]

iocage set resolver="nameserver <IP of nameserver>" <jail>; iocage restart <jail>

I always recommend not messing with ipfw and NAT on TrueNAS but use static routes for VPN networks and the like. It's unsupported and it produces hard to debug side effects - qed.

 

[Aephir]

not messing with ipfw and NAT on TrueNAS but use static routes for VPN networks and the like

I actually remember trying that way back when, but could never get it to work without these tunables. At least not where I also had full access to the rest of my LAN via VPN. So ended up using the tunables, that work pretty much in the first try.

But I also do have a static route:

Code:

destination = 10.10.10.0/24
gateway = 10.0.30.27

"10.10.10.0" is VPN VLAN.
"10.0.30.27" is TrueNAS.

Do you know of a good guide to avoid the firewall and nat in tunables, but still use VPN and have full local access? Then I might just give it another go, it is a long time since I tried.

 

[Patrick M. Hausen]

destination = 10.10.10.0/24 gateway = 10.0.30.27

You need that route - but on your Internet router, not on your TrueNAS. Is that the case? Then you should be able to remove all the firewall related tunables except gateway_enable = yes on the TrueNAS and the VPN should work without interfering with any other jails or VMs.

 

[Aephir]

It's all coming back to me now, what I tried so long ago.

When trying to set a static route, I just get "There was an error updating settings. This action could not be completed.". After "upgrading" (huge quotation marks!) from an Edgerouter to UniFi security gateway, I installed the Home Assistant addon version of the UniFi controller. I then realized this had some serious limitations; I'm guessing this is just another one of them?

These limitations are the reason I wanted to swtich to a "real" unifi controller in TrueNAS jail.

But that means there's a bit of a catch 22; I can't make the static routes needed to switch to UniFi in a TrueNAS jail so I can make the static routes (and other "advanced config").

Maybe I should just find an old Edgerouter and get all funcitonality back, at the expense of the fancy UI with all config in one place :)

 

[Patrick M. Hausen]

If the Unifi controller is in a VNET jail bridged to your main LAN that also connects all your Unifi devices, no static route is needed. You can even distribute the IP address of the controller via DHCP to speed up the adoption of new devices:

It's explained here for Mikrotik, but DHCP is DHCP ...

The static route - again, it needs to go on your Internet router, not the TrueNAS - is only necessary for VPN clients to reach your LAN. It replaces the ipfw/NAT hack.