metallb + BGP = conflict with kube-router?

[s4nguine]

Hello,

I'm planning to move my NAS to TrueNAS and I'm really happy that under the hood of Apps, there is a running K3s since I already have a bare-metal K8s cluster so the knowledge I got there might be useful also on TrueNAS.

In my home I have router on stick setup, to be more specific OPNsense, and I wanted to reflect what I have on different cluster; metallb has assigned one VIP address per pool per VLAN in my setup, and then it's assigned to different Ingress Controllers. It's working just fine.

Now moving to the TrueNAS K3s - I pretty much reflected the setup I have tested, of course using metallb-config app, the address is correctly assigned to the service, I can even enter Traefik dashboard from the same VLAN using LoadBalancer IP but it's not working from different VLANs.

My only guess is that metallb BGP is fighting with kube-router, I can see from the netstat that BGP port 179 is open and assigned to kube-router service and looks like this conflict: docs

Was anyone out there able to deploy metallb with working BGP and L3 routing?
Thanks

 

[s4nguine]

You know what? I'm stupid, the whole time I was looking at VIP, and thought it was needed for Ingress (I'm using one for load balancing API in another cluster...) Well it's not, the only thing I had to do was to configure BGP on both sides and specify any IP range outside of the DHCP. Now everything is working as expected.