TrueNAS 13.0-U1 WireGuard jail - network traffic stops after some time

[Whattteva]

I'll try this procedure; sounds easier than a jail. Will report back. Many thanks for pointing this out.

Good luck! Let me know if you come across any issues. It was fairly simple and painless and I got it working almost immediately when I set it up.

 

[Cellobita]

Good luck! Let me know if you come across any issues. It was fairly simple and painless and I got it working almost immediately when I set it up.

Will do. Thanks again.

 

[Cellobita]

Good luck! Let me know if you come across any issues. It was fairly simple and painless and I got it working almost immediately when I set it up.

Indeed, much easier to set up than the jail method; had it working in under 5 minutes - I've started a continuous ping test to evaluate stability. So far, everything's looking good.

 

[Cellobita]

Good luck! Let me know if you come across any issues. It was fairly simple and painless and I got it working almost immediately when I set it up.

BTW: I couldn't figure out how to allow access to other hosts on the same LAN subnet as the TrueNAS box; can you point me in the right direction?

 

[Whattteva]

BTW: I couldn't figure out how to allow access to other hosts on the same LAN subnet as the TrueNAS box; can you point me in the right direction?

I didn't really require any fancy setups other than specifying the network subnets on the config file. Oh, I also had to add static routes on my main router for the Wireguard subnets.

In my case, the LAN subnet is 192.168.50.0/24 (TrueNAS box is on 192.168.50.10 on the LAN and 10.0.0.1 on the WG LAN) and the WG subnet is 10.0.0.0/24.
I had to add 10.0.0.0/24 static route to my main LAN router and then I had to add 192.168.50.0/24 subnet to the client peer Wireguard config and everything works as expected.

 

[Cellobita]

I didn't really require any fancy setups other than specifying the network subnets on the config file. Oh, I also had to add static routes on my main router for the Wireguard subnets.

In my case, the LAN subnet is 192.168.50.0/24 (TrueNAS box is on 192.168.50.10 on the LAN and 10.0.0.1 on the WG LAN) and the WG subnet is 10.0.0.0/24.
I had to add 10.0.0.0/24 static route to my main LAN router and then I had to add 192.168.50.0/24 subnet to the client peer Wireguard config and everything works as expected.

Didn't work for me: I'm still unable to ping anything on the remote LAN subnet except the WireGuard server/TrueNAS box own IP address.

Anyway, if you're tunneling packets inside the WireGuard connection, why would you need to change the router configuration? That isn't necessary when setting up WireGuard inside a dedicated server, or a TN VM or jail. What am I missing, here?

FWIW, I've tried enabling packet forwarding on the TrueNAS box through System -> Tunables:

Variable=gateway_enable
Value=YES
Type=rc
Description=ip forwarding
Enabled=yes

But it made no difference.

 

[Whattteva]

Didn't work for me: I'm still unable to ping anything on the remote LAN subnet except the WireGuard server/TrueNAS box own IP address.

You can refer to my setup below.

Anyway, if you're tunneling packets inside the WireGuard connection, why would you need to change the router configuration? That isn't necessary when setting up WireGuard inside a dedicated server, or a TN VM or jail. What am I missing, here?

It needs the static route because my local LAN would otherwise have no idea where to route packets destined for the Wireguard LAN, which is a different subnet from the LAN itself. With the static route, it knows to send it to the TrueNAS box. Actually, I have other static routes setup because my home LAN consists of 3 different routers as I will show you later.

FWIW, I've tried enabling packet forwarding on the TrueNAS box through System -> Tunables:



But it made no difference.

Ah yes, it looks like I have this tunable also, but I don't remember if I set it up for the Wireguard or for something else a while ago. In any case, it works so I will leave it on.


Here's my router static routes settings. As you can see, I have 3 routers setup:
- 192.168.50.1 is my main internet gateway.
- 192.168.50.2 -> 192.168.51.0/24 is the router in the bedroom.
- 192.168.50.3 -> 192.168.52.0/24 is the router in the living room.
- 192.168.50.10 -> 10.0.0.0/24 is the Wireguard LAN and 192.168.50.10 is the TrueNAS box.
Also, in this setup, the TrueNAS box is essentially a router for the 10.0.0.0/24 subnet.

Here are my Wireguard server configuration file:

Code:

[Interface]
PrivateKey = <server private key>
Address = 10.0.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <My Android phone pub key>
AllowedIPs = 10.0.0.2/32

[Peer]
PublicKey = <wife's iPhone pub key>
AllowedIPs = 10.0.0.3/32

Here's my Android phone config. 192.168.50.11 is my DNS server for LAN name resolution and ad blocking.

Code:

[Interface]
PrivateKey = <My Android phone private key>
ListenPort = 21841
Address = 10.0.0.2/24
DNS = 192.168.50.11

[Peer]
PublicKey = <server pub key>
Endpoint = <server_address>:51820
AllowedIPs = 10.0.0.0/24, 192.168.50.0/24, 192.168.51.0/24, 192.168.52.0/24

With these network setups, I'm able to ping from/to 10.x.x.x or 192.168.x.x networks in both directions. Most people's setups I'd imagine should be much simpler since most people only have 1 LAN subnet instead of 3 like I do.

 

[Whattteva]

Didn't work for me: I'm still unable to ping anything on the remote LAN subnet except the WireGuard server/TrueNAS box own IP address.

Anyway, if you're tunneling packets inside the WireGuard connection, why would you need to change the router configuration? That isn't necessary when setting up WireGuard inside a dedicated server, or a TN VM or jail. What am I missing, here?

FWIW, I've tried enabling packet forwarding on the TrueNAS box through System -> Tunables:



But it made no difference.

Did it help to get your setup to work?

 

[Cellobita]

Did it help to get your setup to work?

Appreciate your concern, but unfortunately no...

The reason I believe it is not related to any router configuration is that the exact same server config file (wg0.conf) works on a VM or a jail, on the same machine.

I'll keep digging on my spare time, and will post back any new info.

Thanks.

 

[Cellobita]

I may have figured this, at least in part - no definitive solution, though.

Experimenting with tcdump, while pinging from my workstation to another box inside the remote network (at 192.168.1.2), via my TN WG server (at 192.168.1.8), I've found out that the ping source IP is not NAT'ed - it is passed along unchanged with the source IP on my workstation peer config (10.225.30.6):

13:17:08.029844 IP 10.225.30.6 > 192.168.1.2: ICMP echo request, id 1, seq 78, length 40 13:17:08.029897 IP 192.168.1.2 > 10.225.30.6: ICMP echo reply, id 1, seq 78, length 40

However, that box has no idea how to route back to my peer IP, via the WG server, and so the replies are sent to the default gateway and subsequently discarded.

If I manually add a route to this other box (at 192.168.1.2) for the peer subnet, using as a gateway the WG server (192.168.1.8)

route add 10.225.30.0/24 192.168.1.8

Then the ICMP replies get back correctly to my workstation.

This does not happen when WireGuard runs inside a jail or VM: the packets are NAT'ed with the local IP address of the server; compare the example above with the same test while running WG inside a jail, properly translating the source IP on my workstation peer config (10.225.30.6) to the IP on the WG server (192.168.1.8)

13:46:51.227567 IP 192.168.1.8 > 192.168.1.2: ICMP echo request, id 1, seq 83, length 40 13:46:51.228499 IP 192.168.1.2 > 192.168.1.8: ICMP echo reply, id 1, seq 83, length 40

There must be some additional config to enable NAT when using the native WireGuard implementation on TrueNAS, and I'll try to find out the proper commands to do so...

Is my interpretation of the issue correct, or am I missing something?

Edit: I understand that you can add a static route pointing the WG peer subnet [10.225.30.0/24] back to the IP of the WG server [192.168.1.8], but some ISP routers here in Brazil do not allow that - a solution that uses NAT, if indeed possible, works better for me.

 

[Whattteva]

I may have figured this, at least in part - no definitive solution, though.

Experimenting with tcdump, while pinging from my workstation to another box inside the remote network (at 192.168.1.2), via my TN WG server (at 192.168.1.8), I've found out that the ping source IP is not NAT'ed - it is passed along unchanged with the source IP on my workstation peer config (10.225.30.6):

13:17:08.029844 IP 10.225.30.6 > 192.168.1.2: ICMP echo request, id 1, seq 78, length 40 13:17:08.029897 IP 192.168.1.2 > 10.225.30.6: ICMP echo reply, id 1, seq 78, length 40

However, that box has no idea how to route back to my peer IP, via the WG server, and so the replies are sent to the default gateway and subsequently discarded.

If I manually add a route to this other box (at 192.168.1.2) for the peer subnet, using as a gateway the WG server (192.168.1.8)

route add 10.225.30.0/24 192.168.1.8

Then the ICMP replies get back correctly to my workstation.

This does not happen when WireGuard runs inside a jail or VM: the packets are NAT'ed with the local IP address of the server; compare the example above with the same test while running WG inside a jail, properly translating the source IP on my workstation peer config (10.225.30.6) to the IP on the WG server (192.168.1.8)

13:46:51.227567 IP 192.168.1.8 > 192.168.1.2: ICMP echo request, id 1, seq 83, length 40 13:46:51.228499 IP 192.168.1.2 > 192.168.1.8: ICMP echo reply, id 1, seq 83, length 40

There must be some additional config to enable NAT when using the native WireGuard implementation on TrueNAS, and I'll try to find out the proper commands to do so...

Is my interpretation of the issue correct, or am I missing something?

Edit: I understand that you can add a static route pointing the WG peer subnet [10.225.30.0/24] back to the IP of the WG server [192.168.1.8], but some ISP routers here in Brazil do not allow that - a solution that uses NAT, if indeed possible, works better for me.

Sounds correct. This is the reason why I basically have a static route defined on my router for the Wireguard subnet.
What do you exactly mean by ISP router? A router should be a personal device that you own so you should be able to do whatever you want... unless you're leasing a modem/router all-in-one device from your ISP. My ISP offers this too, but I disabled the router functionality cause quite frankly, my router is a much better router than that crappy modem/router device that they loan out.

 

[Cellobita]

Sounds correct. This is the reason why I basically have a static route defined on my router for the Wireguard subnet.
What do you exactly mean by ISP router? A router should be a personal device that you own so you should be able to do whatever you want... unless you're leasing a modem/router all-in-one device from your ISP. My ISP offers this too, but I disabled the router functionality cause quite frankly, my router is a much better router than that crappy modem/router device that they loan out.

On my customers I usually disable the routing functionality from their ISP-provided units (using them basically as ONT's), and pair them with a dedicated, mid-range router; however, quite a few broadband links in Brazil are provided with a combined device [ONT + Router + Wireless AP] that doesn't allow much in terms of customization.

Anyway, I'd rather find a way to enable NAT inside TN - this will allow me to ignore the issues involved in using a substandard router.

 

[EvanVanVan]

@Cellobita Ty for this thread. I was stuck for 24 hours having no clue why Wireguard wasn't working before finding this thread. I had just upgraded from TrueNAS 12.0 to 13.0 and actually at that point was when Wireguard wasn't working with the jails still on 12.0-RELEASE. I upgraded them to 13.0-RELEASE in an attempt to fix them and obviously still had no luck. Finally, after finding this thread, I upgraded them to 13.1-RELEASE and it fixed everything, mostly.

Similar to you, the tunnel seems to be dropping after some time on my Android phone, but fortunately deactivating and reactivating it fixes it for me. But it's also been difficult to troubleshoot/diagnose from my phone not being sure how sketchy the cellular signal is at any point in time. My parents haven't reported the same behavior for them from Windows though which is good.

 

[Cellobita]

The issue is still present with a 13.1-RELEASE jail - the WireGuard server stops responding after a variable amount of time (from a few minutes to several hours).

I have been using TN's native WG support - a lot easier to setup, but for me a less than optimal solution, as I can't easily reach other devices on the remote network (see above).

 

[EvanVanVan]

The issue is still present with a 13.1-RELEASE jail - the WireGuard server stops responding after a variable amount of time (from a few minutes to several hours).

I have been using TN's native WG support - a lot easier to setup, but for me a less than optimal solution, as I can't easily reach other devices on the remote network (see above).

Actually after posting this morning I found this reddit post which describes my current issues. I'm now hoping/wondering if my connection issues are Android client related rather than TN jail/wireguard-server port related.

Just wondering have you enabled the system tunables to try using the wireguard kernel support rather than wireguard-go?

 

[Cellobita]

Yes, I have enabled them - but my remote systems are all running Windows (10 and 11); seems like a different issue than the one on Reddit.

 

[K1LLA_KING_KONG]

What is the latest news on a stable release for wireguard? Is a 13.1 jail the way to go?

 

[Whattteva]

What is the latest news on a stable release for wireguard? Is a 13.1 jail the way to go?

It's built-in with even official guide.

Enabling WireGuard

Provides instructions on setting up WireGuard on TrueNAS CORE.

www.truenas.com

 

[K1LLA_KING_KONG]

It's built-in with even official guide.

Enabling WireGuard

Provides instructions on setting up WireGuard on TrueNAS CORE.

www.truenas.com

ah, Im using this guide: https://www.truenas.com/community/resources/how-to-setup-a-wireguard-vpn-server-in-a-jail.147/

OK ill try that one