I will soon be moving to an environment where I don't control my LAN, and can't have jails with their own IPs.
I enabled NAT on all my jails using the checkbox in the WebUI. Most everything was configured as expected, except there's no internet connectivity in the jails. Upon further inspection, it appears that packets are escaping the NAT just fine, but the replies (eg. DNS) arrive but do not make it onto the bridge.
The ipfw rules look correct (see below) and I would expect the return packets to hit the nat in recv rule.
Am I missing something? Thanks in advance!
Here's my setup:
My LAN (let's say this is out of my control) is 192.168.0.0/24
My FreeNAS's internal LAN is 192.168.1.0/24
Ifconfig from the host outside jails says:
ifconfig from within one of the jail looks like:
Here's the current ipfw setup, as configured automatically by freenas:
Here's what I'm seeing while resolving google.com from inside one of the jails, as monitored by the bridge. Note no replies here:
Here's what the interface sees. Note the proper replies are here:
I enabled NAT on all my jails using the checkbox in the WebUI. Most everything was configured as expected, except there's no internet connectivity in the jails. Upon further inspection, it appears that packets are escaping the NAT just fine, but the replies (eg. DNS) arrive but do not make it onto the bridge.
The ipfw rules look correct (see below) and I would expect the return packets to hit the nat in recv rule.
Am I missing something? Thanks in advance!
Here's my setup:
My LAN (let's say this is out of my control) is 192.168.0.0/24
My FreeNAS's internal LAN is 192.168.1.0/24
Ifconfig from the host outside jails says:
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> ether d0:50:99:1c:32:d7 inet 192.168.0.105 netmask 0xffffff00 broadcast 192.168.0.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (100baseTX <full-duplex>) status: active igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO> ether d0:50:99:1c:32:d8 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (100baseTX <full-duplex>) status: active ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 nd6 options=9<PERFORMNUD,IFDISABLED> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:fe:4a:c8:9c:00 inet 192.168.1.254 netmask 0xffffffff broadcast 192.168.1.254 nd6 options=1<PERFORMNUD> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair9a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 17 priority 128 path cost 2000 member: epair8a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 16 priority 128 path cost 2000 member: epair7a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 15 priority 128 path cost 2000 member: epair6a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 14 priority 128 path cost 2000 member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 13 priority 128 path cost 2000 member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 12 priority 128 path cost 2000 member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000 member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000 member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000 member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000 member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 2 priority 128 path cost 200000 epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:c5:bc:00:08:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:11:3c:00:09:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:fa:d6:00:0a:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:67:89:00:0b:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:e8:f9:00:0c:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair5a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:e1:f9:00:0d:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair6a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:9a:db:00:0e:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair7a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:3c:cb:00:0f:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair8a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:83:cf:00:10:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active epair9a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:82:35:00:11:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active
ifconfig from within one of the jail looks like:
Code:
[root@zetta ~]# jexec 2 ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:38:58:00:0b:0b inet 192.168.1.22 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
Here's the current ipfw setup, as configured automatically by freenas:
Code:
00100 allow ip from any to any via lo0 00200 nat 100 ip from any to 192.168.0.105 in recv igb0 00300 nat 100 ip from 192.168.1.23 to any out xmit igb0 00400 nat 100 ip from 192.168.1.22 to any out xmit igb0 00500 nat 100 ip from 192.168.1.26 to any out xmit igb0 00600 nat 100 ip from 192.168.1.27 to any out xmit igb0 00700 nat 100 ip from 192.168.1.25 to any out xmit igb0 00800 nat 100 ip from 192.168.1.19 to any out xmit igb0 00900 nat 100 ip from 192.168.1.24 to any out xmit igb0 01000 nat 100 ip from 192.168.1.28 to any out xmit igb0 01100 nat 100 ip from 192.168.1.21 to any out xmit igb0 01200 nat 100 ip from 192.168.1.20 to any out xmit igb0 65535 allow ip from any to any
Here's what I'm seeing while resolving google.com from inside one of the jails, as monitored by the bridge. Note no replies here:
Code:
# tcpdump -i bridge0 -vv udp port 53 tcpdump: listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes 14:21:20.688341 IP (tos 0x0, ttl 64, id 6689, offset 0, flags [none], proto UDP (17), length 56) 192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28) 14:21:25.689020 IP (tos 0x0, ttl 64, id 6690, offset 0, flags [none], proto UDP (17), length 56) 192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28) 14:21:30.690015 IP (tos 0x0, ttl 64, id 6691, offset 0, flags [none], proto UDP (17), length 56) 192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28)
Here's what the interface sees. Note the proper replies are here:
Code:
# tcpdump -i igb0 -vv udp port 53 tcpdump: listening on igb0, link-type EN10MB (Ethernet), capture size 65535 bytes 14:21:45.722908 IP (tos 0x0, ttl 63, id 6692, offset 0, flags [none], proto UDP (17), length 56) 192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28) 14:21:45.726879 IP (tos 0x0, ttl 64, id 2593, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142, google.com. A 74.125.239.130 (204) 14:21:45.726884 IP (tos 0x0, ttl 63, id 2593, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142, google.com. A 74.125.239.130 (204) 14:21:46.582113 IP (tos 0x0, ttl 64, id 32949, offset 0, flags [none], proto UDP (17), length 70) 192.168.0.105.30648 > 192.168.0.1.domain: [udp sum ok] 37971+ PTR? 1.0.168.192.in-addr.arpa. (42) 14:21:46.596459 IP (tos 0x0, ttl 64, id 2594, offset 0, flags [DF], proto UDP (17), length 70) 192.168.0.1.domain > 192.168.0.105.30648: [udp sum ok] 37971 NXDomain q: PTR? 1.0.168.192.in-addr.arpa. 0/0/0 (42) 14:21:46.596530 IP (tos 0x0, ttl 64, id 32950, offset 0, flags [none], proto UDP (17), length 72) 192.168.0.105.40794 > 192.168.0.1.domain: [udp sum ok] 37972+ PTR? 105.0.168.192.in-addr.arpa. (44) 14:21:46.610952 IP (tos 0x0, ttl 64, id 2595, offset 0, flags [DF], proto UDP (17), length 72) 192.168.0.1.domain > 192.168.0.105.40794: [udp sum ok] 37972 NXDomain q: PTR? 105.0.168.192.in-addr.arpa. 0/0/0 (44) 14:21:46.611019 IP (tos 0x0, ttl 64, id 32953, offset 0, flags [none], proto UDP (17), length 71) 192.168.0.105.11138 > 192.168.0.1.domain: [udp sum ok] 37973+ PTR? 20.1.168.192.in-addr.arpa. (43) 14:21:46.624888 IP (tos 0x0, ttl 64, id 2596, offset 0, flags [DF], proto UDP (17), length 71) 192.168.0.1.domain > 192.168.0.105.11138: [udp sum ok] 37973 NXDomain q: PTR? 20.1.168.192.in-addr.arpa. 0/0/0 (43) 14:21:50.723028 IP (tos 0x0, ttl 63, id 6693, offset 0, flags [none], proto UDP (17), length 56) 192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28) 14:21:50.724996 IP (tos 0x0, ttl 64, id 2598, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142 (204) 14:21:50.725001 IP (tos 0x0, ttl 63, id 2598, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142 (204) 14:21:55.724022 IP (tos 0x0, ttl 63, id 6694, offset 0, flags [none], proto UDP (17), length 56) 192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28) 14:21:55.725978 IP (tos 0x0, ttl 64, id 2599, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.142, google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134 (204) 14:21:55.725986 IP (tos 0x0, ttl 63, id 2599, offset 0, flags [DF], proto UDP (17), length 232) 192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.142, google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134 (204)