SOLVED Wireguard & Jail access from remote

[oldguy]

I was able to configure wireguard on freenas and i am able to reach its internal IP, however i am not able to reach nextcloud jail ip, i am new to freebsd any help much appreciated,

Pleas note that wireguard is not in jail, so what do we have here

WG0 interface
Client1 10.66.66.3 > 10.66.66.1 Freenas

Local network
Freenas 192.168.2.46
Nextcloud jail 192.168.2.48

A client is able to ping from 10.66.66.3 to 192.168.2.46 but not the 192.168.2.48(nextcloud jail), when i tcpdump the interface i see the ICMP coming, but no reply:

11:38:09.793698 IP 10.66.66.3 > 192.168.2.48: ICMP echo request, id 1, seq 12, length 40
11:38:14.460278 IP 10.66.66.3 > 192.168.2.48: ICMP echo request, id 1, seq 13, length 40

Same ping to Nas interface, no problem:

11:38:01.631401 IP 10.66.66.3 > 192.168.2.46: ICMP echo request, id 1, seq 9, length 40
11:38:01.631411 IP 192.168.2.46 > 10.66.66.3: ICMP echo reply, id 1, seq 9, length 40
11:38:02.636318 IP 10.66.66.3 > 192.168.2.46: ICMP echo request, id 1, seq 10, length 40
11:38:02.636327 IP 192.168.2.46 > 10.66.66.3: ICMP echo reply, id 1, seq 10, length 40


I have enabled net.inet.ip.forwarding: 1
and my routing table looks like this re0 is my defaut interface

default 192.168.2.254 UGS re0
10.66.66.1 link#5 UH wg0
10.66.66.2/32 wg0 US wg0
10.66.66.3/32 wg0 US wg0
localhost link#2 UH lo0
172.16.0.1/32 wg0 US wg0
192.168.2.0/24 link#1 U re0
192.168.2.46 link#1 UHS lo0

Ifconfig looks like this:

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether 34:64:a9:35:30:ae
hwaddr 34:64:a9:35:30:ae
inet 192.168.2.46 netmask 0xffffff00 broadcast 192.168.2.255
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:12:11:0c:6c:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000
member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: Nextcloud as nic: epair0b
options=8<VLAN_MTU>
ether 34:64:a9:27:9a:bf
hwaddr 02:e7:d0:00:04:0a
inet6 fe80::3664:a9ff:fe27:9abf%vnet0.1 prefixlen 64 scopeid 0x4
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 10.66.66.1 --> 10.66.66.1 netmask 0xffffff00
nd6 options=101<PERFORMNUD,NO_DAD>
groups: tun
Opened by PID 1901


I am not really sure what to do next, how to make in sort that i can reach the nextcloud jail ?

Thank you

 

[oldguy]

Hello, looks like part of my copy paste was eaten, and looks like i can not edit my post; so it starts like this:
-----------------------------------------------------------------------------------------------------------------
Hello, i have installed freenas recently, and its a great sotfware, ruen into one problem with internal networking

 

[Kcaj]

I was able to configure wireguard on freenas and....
~
Pleas note that wireguard is not in jail...

Its not recommended to install anything on FreeNAS itself, please rethink your strategy.
I am also yet to come across a situation which I couldn't configure networking though the GUI.

Please use </> Code tags around your code.

 

[oldguy]

Thank you for your reply, i have followed the blog post: https://www.ixsystems.com/blog/wireguard-on-freenas-11-3 they dont suggest to install the package in jail, i must say that i am pretty new to freebsd, so i just followed the guide, the only information i have found is that a jail must use a routing table of the host, but looks like it is not in this case here, as the host has the correct route to the VPN IP's
I have noted the suggestion for the "code" i can not edit the previous post, will use for next message for sure.

 

[Kcaj]

Thank you for your reply, i have followed the blog post: https://www.ixsystems.com/blog/wireguard-on-freenas-11-3 they don't suggest to install the package in jail...

If didn't know about this, thanks!

From your ifconfig your bridge interface has two members re0 and vnet0.1, so any traffic for your jail would be using this interface.

Did you manually configure the 10.66.66.3/32 wg0 US wg0 route?

 

[oldguy]

The wg0 interface is configured via wg0.conf file which is read by wireguard daemon:

Code:

[Interface]
Address = 10.66.66.1/24
ListenPort = 10000
PrivateKey = 0AHeDL7f/tVULAWkWHqO3Y=

[Peer]
PublicKey = VqkwSgTKit3ENPloum0HowM=
AllowedIPs = 10.66.66.2/32,172.16.0.1/32

[Peer]
PublicKey = 2h7EVjZVWxV6wVMTlTUU4=
AllowedIPs = 10.66.66.3/32

Routes are configured by wireguard daemon accordinly the PEER declaration

I copy paste again the routes here for a good visibility:

Code:

Destination        Gateway            Flags     Netif Expire
default            192.168.2.254      UGS         re0
10.66.66.1         link#5             UH          wg0
10.66.66.2/32      wg0                US          wg0
10.66.66.3/32      wg0                US          wg0
localhost          link#2             UH          lo0
172.16.0.1/32      wg0                US          wg0
192.168.2.0/24     link#1             U           re0
192.168.2.46       link#1             UHS         lo0

also the ifconfig again:

Code:

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 34:64:a9:35:30:ae
        hwaddr 34:64:a9:35:30:ae
        inet 192.168.2.46 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:12:11:0c:6c:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: Nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether 34:64:a9:27:9a:bf
        hwaddr 02:e7:d0:00:04:0a
        inet6 fe80::3664:a9ff:fe27:9abf%vnet0.1 prefixlen 64 scopeid 0x4
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
        options=80000<LINKSTATE>
        inet 10.66.66.1 --> 10.66.66.1 netmask 0xffffff00
        nd6 options=101<PERFORMNUD,NO_DAD>
        groups: tun
        Opened by PID 1901

So a VPN client with 10.66.66.3 can ping 192.168.2.46 Freenas, but not 192.168.2.48 (nextcloud jail)

I have checked, the Client, has correctly network 192.168.2.0/24 routed by his configuration, and ping to nextcloud IP comes to wg0 interface, but i am not getting a reply from 192.168.2.48 back to WG0

 

[Kcaj]

..ping to nextcloud IP comes to wg0 interface, but i am not getting a reply from 192.168.2.48 back to WG0

Is the ping actually being received by the bridge0 or vnet0.1 interfaces though? (I dont think so)
I assume you can reach the nextcolud from the re0 interface?

I am not sure how you would go about this, you could try add the wg0 as a member of bridge0 though I dont think it would work being L2. You get a response from the 192.168.2.46 because is has a configured route in the routing table, you could so try add a static route to your jail, though someone smarter than me should probably help you if thats the case.

 

[oldguy]

I was thinkin like in worst case my ping would go to default route, bounce to my switch and go back to nextcloud per switch ARP table.

So i have dumped the bridge0 and looks like Nextcloud received the PING from the VPN client, and gives a reply:

Code:

07:31:58.379797 IP 192.168.2.48 > 10.66.66.3: ICMP echo reply, id 1, seq 35, length 40
07:32:03.406383 IP 192.168.2.48 > 10.66.66.3: ICMP echo reply, id 1, seq 36, length 40
07:32:08.364623 IP 192.168.2.48 > 10.66.66.3: ICMP echo reply, id 1, seq 37, length 40

So the problem is the reply route

 

[oldguy]

I would like to clarify that this ping is not present at wg0 interface, i am not sure where to dig now, i have pinged Client IP from freenas, no problem, which means that inside freenas the routing table is working fine, but something is blocking the reply from Jail to external ip.
From inside network without VPN i can reach everyone.

 

[Kcaj]

I belive your right that your ping is going to your router and then back, both via your re0 interface. Your router wont know the route to your VPN client since its a static route on your FreeNAS.

Can you monitor yourre0?

Edit: you can also use the route show <ip address> to see how your FreeNAS is making routing decisions....

 

[oldguy]

On the re0 interface i see a reply from nextclud jail 192.168.2.48 to 10.66.66.3 so looks like you are correct

Code:

09:45:16.453436 IP 192.168.2.48 > 10.66.66.3: ICMP echo reply, id 1, seq 153, length 40
09:45:21.146283 IP 192.168.2.48 > 10.66.66.3: ICMP echo reply, id 1, seq 154, length 40

But this is strange as per host routing table the default GW is the last, i am missing something

Is there any way to change that ? I tried to setup freenas as GW, without effect.
How to make in sort that the routing table of the host is used ?

 

[Kcaj]

But this is strange as per host routing table the default GW is the last, i am missing something

Your default gateway is 192.168.2.254 which is found via interface re0.

Maybe the easiest would be to attach the jail to the wg0 interface and put it in the 10.66.66.0 network but then it wont be available on your normal LAN.

or
tell your router where the 10.66.66.0 network is.

 

[oldguy]

Hello, so i solved the problem by learning how to jexec into jail, then i looked in to the routing table:

Code:

Destination        Gateway            Flags     Netif Expire
default            192.168.2.254      UGS     epair0b
localhost          link#1             UH          lo0
192.168.2.0/24     link#2             U       epair0b
192.168.2.48       link#2             UHS         lo0

As you see, the information i have found that jail is getting the roting table from host is totally false, so i have added a route

route add -net 10.66.66.0/24 192.168.2.46

And imediately all worked fine, as the ipv4 routing was activated.
I would like to thank you for help, without you i would not be possible to find the solution.
Next, i need to add that route on boot of the jail, and it would be it.
That get me thinking that i wireguard was in jail the same way, i think it would still be required to add routes back and forth.
Have a nice day!

 

[oldguy]

Next week i hope i can share a speed test from remote location, both sites have 1000Mbps download, and 600Mbps upload, should be faster in theory.

 

[elrodvt]

I'm not much at networking and still at a loss as to how to have my jails traffic sent through the wireguard wg0 interface when it's active. I'm on 11.3 with plugin configured jails that look pretty much like what @oldguy showed. All jails can connect to each other and the private label and internet. Once wg is started though the jails can talk between themselves or to the freenas base is but that's it.
@oldguy could you show your routing tables for both a jail and the freenas host (wireguard client) now that you have it working? A traceroute to Google or a route show output to them might help also.
Thanks in advance!

 

[elrodvt]

Note, I meant private lan above which for me is good old 192.168.0.1

 

[oldguy]

routing table in jail:

Code:

Destination        Gateway            Flags     Netif Expire
default            192.168.2.254      UGS     epair0b
10.66.66.0/24      192.168.2.46       UGS     epair0b
localhost          link#1             UH          lo0
192.168.2.0/24     link#2             U       epair0b
192.168.2.48       link#2             UHS         lo0

Routing table in host:

Code:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.2.254      UGS         re0
10.66.66.1         link#3             UH          wg0
10.66.66.2/32      wg0                US          wg0
10.66.66.3/32      wg0                US          wg0
localhost          link#2             UH          lo0
172.16.0.1/32      wg0                US          wg0
192.168.2.0/24     link#1             U           re0
192.168.2.46       link#1             UHS         lo0

script to add route on boot:

Code:

#!/bin/sh

. /etc/rc.subr

name=wireguard_enable
rcvar=wireguard_enable

start_cmd="${name}_start"
stop_cmd=":"

load_rc_config $name

wireguard_enable_start()
{
/sbin/route add -net 10.66.66.0/24 192.168.2.46
}

run_rc_command "$1"

i have also enabled the ipv4 routing in freebsd otherwise it will not work

 

[elrodvt]

Thanks, I'll give that a whirl.

 

[elrodvt]

@oldguy still no luck. When you say "i have also enabled the ipv4 routing in freebsd"
does that mean adding gateway_enable="YES" to the tunables for rc.d? Sorry to require so much force feeding. ;-)

 

[elrodvt]

Follow up:

freenas host netstat -r
Destination Gateway Flags Netif Expire
0.0.0.0/1 wg0 US wg0
default 192.168.0.1 UGS igb1
10.23.5.0/24 192.168.0.82 UGS igb1
10.23.5.3 link#7 UH wg0
localhost link#3 UH lo0
128.0.0.0/1 wg0 US wg0
169.254.0.0/16 link#1 U igb0
169.254.1.9 link#1 UHS lo0
172.16.0.0/30 link#4 U vnet0.1
172.16.0.1 link#4 UHS lo0
172.16.0.4/30 link#5 U vnet0.2
172.16.0.5 link#5 UHS lo0
172.16.0.8/30 link#6 U vnet0.3
172.16.0.9 link#6 UHS lo0
172.16.0.12/30 link#8 U vnet0.9
172.16.0.13 link#8 UHS lo0
192.168.0.0/24 link#2 U igb1
192.168.0.82 link#2 UHS lo0

jail (for jail 172.16.0.2)

route add -net 10.23.5.0/24 192.168.0.82

netstat -r

Internet:
Destination Gateway Flags Netif Expire
default 172.16.0.1 UGS epair0b
localhost link#1 UH lo0
172.16.0.0/30 link#2 U epair0b
172.16.0.2 link#2 UHS lo0

when wg is down:
all jails go to the internet via igb0 and nat sends back to the jail based on it's port.
all jails can ping each other and the 192.168.0 local lan.

when wg is up:
all jails internet bound traffic either doesn't go out or doesn't come back. not sure which.
all jails can ping each other and the 192.168.0 local lan.