Kubernetes IP assignment Question/Issue.

[guyp2k]

Question, I spun up a few pods (sabnazb, lidarr, etc., stack). The issue I have is that I have a FW rule that routes specific hosts through a VPN (wg0). I have assigned specific IPs to those pods and work fine however, the FW still thinks they are sourcing from the host (TrueNAS SCALE), resulting in not routing through VPN/wg0). I assume this has something to do w/ the internal network on kubernetes. Still trying to adjust/learn, any suggestions moving forward?

Also, I can access the apps via the static IP defined on the pods, but my FW thinks they are sourcing from the host/TrueNAS Scale and will not route correctly over the VPN/wg0 interface. Of course if I add the host/TrueNAS SCALE to the FW alias for routing over the VPN/wg0 that works just fine, just the pods will not, even though they have a specific IP on the same subnet and defined in the FW alias.....

 

[ornias]

This is actually a good question...
The easiest way of doing this would be using Apps (aka Helm Charts) that are preconfigured to add outgoing VPN routing with an extra container on the App itself. k8s-at-home allow for this for example.

For TrueCharts i'm planning to add this once we are reasonably certain our current stack, docks and ingress are rocksolid. So 21.08 or 21.10.

 

[cephyr]

This is actually a good question...
The easiest way of doing this would be using Apps (aka Helm Charts) that are preconfigured to add outgoing VPN routing with an extra container on the App itself. k8s-at-home allow for this for example.

For TrueCharts i'm planning to add this once we are reasonably certain our current stack, docks and ingress are rocksolid. So 21.08 or 21.10.

Wow thanks, I am really looking forward to this!!
facing the same problem - currently i have to route my whole truenas box via vpn and define excludes for specific targets

is this already implemented?

 

[truecharts]

Wow thanks, I am really looking forward to this!!
facing the same problem - currently i have to route my whole truenas box via vpn and define excludes for specific targets

is this already implemented?

Its generally frowned upon to "necro" old posts from months ago.

If you've questions about which features we do or do-not support, it's best to checkout our website first and ask directly at our support discord if it's not on the website :)

 

[murzik]

Question, I spun up a few pods (sabnazb, lidarr, etc., stack). The issue I have is that I have a FW rule that routes specific hosts through a VPN (wg0). I have assigned specific IPs to those pods and work fine however, the FW still thinks they are sourcing from the host (TrueNAS SCALE), resulting in not routing through VPN/wg0). I assume this has something to do w/ the internal network on kubernetes. Still trying to adjust/learn, any suggestions moving forward?

Also, I can access the apps via the static IP defined on the pods, but my FW thinks they are sourcing from the host/TrueNAS Scale and will not route correctly over the VPN/wg0 interface. Of course if I add the host/TrueNAS SCALE to the FW alias for routing over the VPN/wg0 that works just fine, just the pods will not, even though they have a specific IP on the same subnet and defined in the FW alias.....

https://jira.ixsystems.com/browse/NAS-112334

Vote for the issue, maybe will get resolved sooner.

 

[DirkTas67]

I guess this is still not resolved. I run into this issue too @murzik I want exactly the same as you.
I'm a new user of Truenas and Pfsense, I setup the VPN on the firewall following the instructions from Tom at L Systems and now I wanted to have apps like Qbittorrent, Jackett.... to go out true the VPN, so I gave them all static IP's and put these IP's in the rule to go out true the VPN, but this still doesn't work, it's only when you add the IP of the server that everything goes true the VPN, and that gives other problems, things like syncthing don't like that.

 

[murzik]

https://jira.ixsystems.com/browse/NAS-115206

I see no progress. That is very sad.

 

[truecharts]

https://jira.ixsystems.com/browse/NAS-115206

I see no progress. That is very sad.

It's pretty uncalled for to complain just 1 week after a low-priority issue is reported on a issuetracker.
The fix time for these minor issues is usually in the weeks-to-months range.

(and yes, it's a minor bug. Maybe not for your, but from a technical perspective it is..)

 

[murzik]

a week?

https://jira.ixsystems.com/browse/NAS-112334

And no one is complaining.

 

[DirkTas67]

It's pretty uncalled for to complain just 1 week after a low-priority issue is reported on a issuetracker.
The fix time for these minor issues is usually in the weeks-to-months range.

(and yes, it's a minor bug. Maybe not for your, but from a technical perspective it is..)

didn't know this was a bug until earlier this week and I didn't want to sound like I was complaining. Good that it is classified as a bug and will be fixed, thank you for the great work!

 

[aherbjornsen]

a week?

https://jira.ixsystems.com/browse/NAS-112334

And no one is complaining.

The issue was closed in December, because the GUI was modified to allow for 0.0.0.0/0 to be added as static route for the container.
Apparently no one bothered to check if this actually worked, and it does not. The container sticks to default route through 172.x.x.x (Kubernetes) interface.

I made a clone of the original issue. Don't know if that is the approved action though.

 

[aherbjornsen]

Forgot to add link to new issue: https://jira.ixsystems.com/browse/NAS-116775

 

[murzik]

Forgot to add link to new issue: https://jira.ixsystems.com/browse/NAS-116775

I have ticket open for this issues since march. Provided all troubleshooting steps that were requested...

https://jira.ixsystems.com/browse/NAS-115206

 

[aherbjornsen]

I have ticket open for this issues since march. Provided all troubleshooting steps that were requested...

https://jira.ixsystems.com/browse/NAS-115206

Yes I see that now. I looked at the last ticket referenced.

Anyway, your thread is strange - like the person you are communicating with does not understand the issue.

There's actually two problems here:
a) There's no way to disable the default route to the 172.x.x.x address
b) The 0.0.0.0/0 route (new default) is not added to the container.

In Linux (at least not on my assorted Ubuntu servers) it is perfectly ok to have more than one 'default'/0.0.0.0/0 route , so both these issues must be sorted.

 

[murzik]

Unfortunately ticket is closed as unresolved. If you are interested read the conversation in Jira.
Explanation seems ridiculous to me, but unfortunately that is all we can get.

 

[aherbjornsen]

Unfortunately ticket is closed as unresolved. If you are interested read the conversation in Jira.
Explanation seems ridiculous to me, but unfortunately that is all we can get.

That is unfortunate, and this does not make sense to me either.
If you can add static routes, you should be able to remove existing routes also. That issue looks like the guy handling it did not really understand the problem.

Anyway, for my use I mitigated the problem by installing a container with both VPN and Transmission - making sure at least my bittorrents are going through my VPN. Not ideal, and annoying if you have several containers you want to tunnel through VPN.

 

[murzik]

As a work around I am using docker compose by Truecharts. with all containers running inside docker compose. Network management works well inside docker compose.

 

[DirkTas67]

This is certainly a pitty, I wanted to have the vpn on the pfsense firewall because it is specialized in this and I'm sure with the right set of rules that no defined device/app will escape the vpn. With the vpn on truenas it is again a black hole for me.