SOLVED Jails and VLANs

dak180 · Jul 15, 2019

So here is what I am trying to do: there are three vlans coming into freenas (10, 50, 60) and there are two jails running, one on vlan 10 (where the web gui also runs) and the other on vlan 60; vlan 50 is mostly unused aside from also being what the bmc is on. This is mostly working; I added the the vlans under networking, added static routes for physical lan and the two vlans in use and set vlan 10 to dhcp (static assignment) and the others to static ips. The jails each have different bridges specified and the apropreate vlan for the vnet default interface, they are also each set to use dhcp which is statically assigned.

I can ping all of the freenas's addresses from my computer (which is on vlan 10) and the jail on vlan 10, but not the one on vlan 60 (there are firewall rules that should allow this), and when I ssh in I cannot ping my computer from the jail on vlan 60. Also I cannot connect to the web management interface on the jail running on vlan 60.

For the firewall I am running pfsense and at the moment each network has a rule passing traffic to the other.

I am hoping someone might be able to tell me how I can make this work or what other info might be useful in diagnosing the problem.

Code:

$ ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:c1:3d:03
    hwaddr d0:50:99:c1:3d:03
    inet 192.168.4.115 netmask 0xffffff00 broadcast 192.168.4.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:c1:3d:04
    hwaddr d0:50:99:c1:3d:04
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether d0:50:99:c1:3d:03
    inet 192.168.60.115 netmask 0xffffff00 broadcast 192.168.60.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether d0:50:99:c1:3d:03
    inet 192.168.9.115 netmask 0xffffff00 broadcast 192.168.9.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 10 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan12: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:c1:3d:03
    inet 192.168.50.115 netmask 0xffffff00 broadcast 192.168.50.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 50 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:6f:4f:03:36:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 20000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether b2:13:dd:98:4a:80
    hwaddr 02:ba:d0:00:08:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:6f:4f:03:36:0b
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 20000
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: transmission as nic: epair0b
    options=8<VLAN_MTU>
    ether 4a:3a:78:77:16:83
    hwaddr 02:ba:d0:00:0a:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

rob_anybody · Jul 16, 2019

Hi
the first thing I would try is to ping your jails from the Firewall.
Next you could traceroute from your pc.

dak180 · Jul 19, 2019

rob_anybody said:
the first thing I would try is to ping your jails from the Firewall.
Next you could traceroute from your pc.

vlan 60:
firewall:

Code:

: ping -c 4 192.168.60.116
PING 192.168.60.116 (192.168.60.116): 56 data bytes
64 bytes from 192.168.60.116: icmp_seq=0 ttl=64 time=0.444 ms
64 bytes from 192.168.60.116: icmp_seq=1 ttl=64 time=1.719 ms
64 bytes from 192.168.60.116: icmp_seq=2 ttl=64 time=0.343 ms
64 bytes from 192.168.60.116: icmp_seq=3 ttl=64 time=0.368 ms

--- 192.168.60.116 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.343/0.719/1.719/0.579 ms

Code:

: traceroute 192.168.60.116
traceroute to 192.168.60.116 (192.168.60.116), 64 hops max, 40 byte packets
 1  transmission (192.168.60.116)  0.489 ms  0.463 ms  0.395 ms

my comp:

Code:

$ ping -c 4 192.168.60.116
PING 192.168.60.116 (192.168.60.116): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 192.168.60.116 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

Code:

traceroute to 192.168.60.116 (192.168.60.116), 64 hops max, 72 byte packets
 1  192.168.9.1 (192.168.9.1)  2.901 ms  1.430 ms  1.299 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *
61  * * *
62  * * *
63  * * *
64  * * *

vlan 10:
firewall:

Code:

: traceroute 192.168.9.117
traceroute to 192.168.9.117 (192.168.9.117), 64 hops max, 40 byte packets
 1  Plex (192.168.9.117)  0.286 ms  0.250 ms  0.276 ms

Code:

: ping -c 4 192.168.9.117
PING 192.168.9.117 (192.168.9.117): 56 data bytes
64 bytes from 192.168.9.117: icmp_seq=0 ttl=64 time=0.301 ms
64 bytes from 192.168.9.117: icmp_seq=1 ttl=64 time=0.291 ms
64 bytes from 192.168.9.117: icmp_seq=2 ttl=64 time=0.409 ms
64 bytes from 192.168.9.117: icmp_seq=3 ttl=64 time=0.389 ms

--- 192.168.9.117 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.291/0.347/0.409/0.052 ms

my comp:

Code:

$ ping -c 4 192.168.9.117
PING 192.168.9.117 (192.168.9.117): 56 data bytes
64 bytes from 192.168.9.117: icmp_seq=0 ttl=64 time=2.330 ms
64 bytes from 192.168.9.117: icmp_seq=1 ttl=64 time=2.822 ms
64 bytes from 192.168.9.117: icmp_seq=2 ttl=64 time=3.394 ms
64 bytes from 192.168.9.117: icmp_seq=3 ttl=64 time=6.633 ms

--- 192.168.9.117 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.330/3.795/6.633/1.681 ms

Code:

$ traceroute 192.168.9.117
traceroute to 192.168.9.117 (192.168.9.117), 64 hops max, 52 byte packets
 1  plex (192.168.9.117)  3.506 ms  1.332 ms  4.017 ms

And as a bonus: from my computer to freenas' vlan 60 interface address:

Code:

$ ping -c 4 192.168.60.115
PING 192.168.60.115 (192.168.60.115): 56 data bytes
64 bytes from 192.168.60.115: icmp_seq=0 ttl=64 time=3.436 ms
64 bytes from 192.168.60.115: icmp_seq=1 ttl=64 time=3.163 ms
64 bytes from 192.168.60.115: icmp_seq=2 ttl=64 time=2.635 ms
64 bytes from 192.168.60.115: icmp_seq=3 ttl=64 time=3.202 ms

--- 192.168.60.115 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.635/3.109/3.436/0.293 ms
$ traceroute 192.168.60.115
traceroute to 192.168.60.115 (192.168.60.115), 64 hops max, 52 byte packets
 1  192.168.9.1 (192.168.9.1)  1.759 ms  1.757 ms  1.326 ms
 2  192.168.60.115 (192.168.60.115)  4.147 ms  2.892 ms  2.323 ms

rob_anybody · Jul 23, 2019

So now we know you can reach the jail from your firewall. Next lets take a look what happens when you try to connect via your pc. You get stuck at 192.168.9.1. Is that your Firewall? If thats the case then your Firewall does not route between the vlans. Else your pc is missing a route.

dak180 · Jul 23, 2019

rob_anybody said:
You get stuck at 192.168.9.1. Is that your Firewall? If thats the case then your Firewall does not route between the vlans. Else your pc is missing a route.

192.168.9.1 is one of my firewall's addresses but as you can see from the last ping / traceroute above my computer can get to the other vlan just not the jail its self.

rob_anybody · Jul 24, 2019

Oh I missed that funfact.
So I could think of some things that could be wrong. As you can ping your vlan60/vlan11 interface from a different Network but not your Jail it could have a wrong default gateway or simply missing one.

So take a look at ifconfig and netstat -r inside the jail.

Now the other thing that could go wrong is vlan tagging. You could use tcpdump to check.

Or you could create a vlan 60 interface inside your Jail and bind the jails ip to it.

dak180 · Jul 28, 2019

rob_anybody said:
So take a look at ifconfig and netstat -r inside the jail.

Code:

# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 4a:3a:78:77:16:82
    hwaddr 02:ba:d0:00:0a:0b
    inet 192.168.60.116 netmask 0xffffff00 broadcast 192.168.60.255
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    inet 10.0.10.6 --> 10.0.10.5 netmask 0xffffffff
    nd6 options=1<PERFORMNUD>
    groups: tun
    Opened by PID 39333

Code:

# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          10.0.10.5          UGS        tun0
default            192.168.60.1       UGS     epair0b
10.0.10.1/32       10.0.10.5          UGS        tun0
10.0.10.5          link#3             UH         tun0
10.0.10.6          link#3             UHS         lo0
91.132.136.43/32   192.168.60.1       UGS     epair0b
127.0.0.1          link#1             UH          lo0
128.0.0.0/1        10.0.10.5          UGS        tun0
192.168.60.0/24    link#2             U       epair0b
192.168.60.116     link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

rob_anybody said:
Now the other thing that could go wrong is vlan tagging. You could use tcpdump to check.

What exactly could be going wrong with the vlan tagging that I would be looking for?

rob_anybody · Jul 29, 2019

dak180 said:
What exactly could be going wrong with the vlan tagging that I would be looking for?

It could be that packages leave the interface untagged.

you could check where your ping reply/request gets lost by using tcpdump.
Just check at your interfaces by using -i and filter for vlan packages and icmp on your freenas while pinging your jail from your pc.

tcpdump -i bridge11 -e vlan and icmp

you should see request and reply packages in vlan 60.

With your current setup it might be that you need to remove the vlan filter inside the jail

tcpdump -i epair0b -e icmp

to see the "ICMP echo request" / "ICMP echo reply" packets

rob_anybody · Jul 29, 2019

rob_anybody said:
Or you could create a vlan 60 interface inside your Jail and bind the jails IP to it.

like so:
ifconfig epair0b delete 192.168.60.116
ifconfig epair0b.60 create vlan 60 vlandev epair0b inet 192.168.60.116/24

dak180 · Jul 29, 2019

rob_anybody said:
tcpdump -i epair0b -e icmp

to see the "ICMP echo request" / "ICMP echo reply" packets

Code:

$ ping -c 4 192.168.60.116
PING 192.168.60.116 (192.168.60.116): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 192.168.60.116 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

Code:

# tcpdump -i epair0b -e icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on epair0b, link-type EN10MB (Ethernet), capture size 262144 bytes
13:20:54.596598 40:62:31:07:d6:7e (oui Unknown) > 4a:3a:78:77:16:82 (oui Unknown), ethertype IPv4 (0x0800), length 98: klimad-rMBP.local.dak180.com > transmission.local.dak180.com: ICMP echo request, id 38858, seq 0, length 64
13:20:55.601458 40:62:31:07:d6:7e (oui Unknown) > 4a:3a:78:77:16:82 (oui Unknown), ethertype IPv4 (0x0800), length 98: klimad-rMBP.local.dak180.com > transmission.local.dak180.com: ICMP echo request, id 38858, seq 1, length 64
13:20:56.600077 40:62:31:07:d6:7e (oui Unknown) > 4a:3a:78:77:16:82 (oui Unknown), ethertype IPv4 (0x0800), length 98: klimad-rMBP.local.dak180.com > transmission.local.dak180.com: ICMP echo request, id 38858, seq 2, length 64
13:20:57.605505 40:62:31:07:d6:7e (oui Unknown) > 4a:3a:78:77:16:82 (oui Unknown), ethertype IPv4 (0x0800), length 98: klimad-rMBP.local.dak180.com > transmission.local.dak180.com: ICMP echo request, id 38858, seq 3, length 64
^C
4 packets captured
9071 packets received by filter
0 packets dropped by kernel

rob_anybody · Jul 29, 2019

Now if you have not figured it out yet.
Take a look at the output of:
route show 192.168.9.0
inside your jail.

dak180 · Jul 30, 2019

rob_anybody said:
Now if you have not figured it out yet.
Take a look at the output of:
route show 192.168.9.0
inside your jail.

It gives no route and the tun0 routes are overriding the default route so route add -net 192.168.0.0/16 192.168.60.1 gets things working but how can this be set (ideally in iocage) to happen automatically?

rob_anybody · Jul 31, 2019

right
so I don't think there is an option to set it via iocage, so I would add it to the jails rc.conf
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-routing.html

dak180 · Aug 2, 2019

It seems like the right option is to add it to the openvpn config:
route 192.168.0.0 255.255.0.0 net_gateway
since the default route will be sufficient so long as the routes added by the vpn are not present.

Important Announcement for the TrueNAS Community.

SOLVED Jails and VLANs

dak180

Patron

rob_anybody

Dabbler

dak180

Patron

rob_anybody

Dabbler

dak180

Patron

rob_anybody

Dabbler

dak180

Patron

rob_anybody

Dabbler

rob_anybody

Dabbler

dak180

Patron

rob_anybody

Dabbler

dak180

Patron

rob_anybody

Dabbler

dak180

Patron

Similar threads