SOLVED Cannot get internet connectivity from jails

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
Hello everyone,

I've had a FreeNAS instance up and running for a few months without any issues, and was running a single plugin (plex) on it. The plugin stopped working after I upgraded to FreeNAS-11.3-U2.1 - it happens. Despite my best efforts, I was never able to bring it back up and ended up deleting it in order to re-install it from scratch, which didn't work either. After trying out various network configurations and struggling with the various error messages, I ended up trying to do the setup manually using danb35's script. This also didn't work, as it turned out that the jail was unable to resolve "pkg.freebsd.org" during the installation process. Investigating further, I created a simple iocage and realized that I had zero connectivity inside of it. I'm fairly experienced with Debian-based systems, but not so much with FreeBSD and simply can't figure out what is going on here - there's an obvious networking problem but it eludes me and I'd appreciate any help in solving it.
I'm assuming that the various failure messages I got during the plugin installation are, one way or another, related to the iocage not being able to access the network.

I'm running FreeNAS on a VM with 8 GB of RAM, over a home-ESX that has two Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz and over 140 GB of RAM. Everything takes place in a VLAN using 10.12.0.0/16 address space:

Code:
Gateway: 10.12.0.1 (also a DHCP and DNS server)
FreeNAS: 10.12.100.3
Jail:    10.12.100.4


If I set up a jail to use DHCP, the following error happens on startup (and the jail doesn't start):
Code:
Error: [EFAULT] + Acquiring DHCP address: FAILED, address received: 0.0.0.0/8 Stopped DHCP_TEST due to DHCP failure
CLOSE


If I set a static IP (10.12.100.4) the jail starts, can ping 10.12.100.3 properly but nothing else (in particular, not the gateway). From the FreeNAS machine, I can ping machines from the network (including the gateway) without any problem. Here is the output of ifconfig for FreeNAS and also from the jail:

Code:
root@freenas[~/]# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em0
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:a1:f5:95
        hwaddr 00:0c:29:a1:f5:95
        inet 10.12.100.3 netmask 0xffff0000 broadcast 10.12.255.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:99:dc:dc:c8:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0.9: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: plexmediaserver as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:0c:29:3d:6b:aa
        hwaddr 02:3d:d0:00:04:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair


Code:
root@plexmediaserver:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:0c:29:3d:6b:ab
        hwaddr 02:a1:5c:00:05:0b
        inet 10.12.100.4 netmask 0xffff0000 broadcast 10.12.255.255
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair


Does anyone see an issue with this?
Thanks in advance for your help.
 
Last edited:

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
Your default gateway is on another subnet.

Edit: Just reread you are using /16 subnet, whats the output of your routing tables netstat -r?

Also post he output of iocage get all <Jail Name> as the 0.0.0.0 on epair0b looks odd to me.
 
Last edited:

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
Here is the output of the requested commands. The first one is run from the iocage, the second one from the FreeNAS.

Code:
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.12.0.1          UGS     epair0b
10.12.0.0/16       link#2             U       epair0b
10.12.100.4        link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0


Code:
root@freenas[~]# iocage get all plexmediaserver
CONFIG_VERSION:26
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:1
boot:0
bpf:1
children_max:0
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:10.12.0.1
defaultrouter6:auto
depends:none
devfs_ruleset:6
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:plexmediaserver
host_hostuuid:plexmediaserver
host_time:1
hostid:602edeae-24e2-11ea-8183-000c29a1f595
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|10.12.100.4/16
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/plexmediaserver/data
jail_zfs_mountpoint:none
last_started:2020-05-03 11:09:22
localhost_ip:none
login_flags:-f root
mac_prefix:000c29
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:11.3-RELEASE-p7
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:000c293d6baa 000c293d6bab
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off
root@freenas[~]# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.12.0.1          UGS         em0
10.12.0.0/16       link#1             U           em0
10.12.100.3        link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
You likley can ping anything without allow_raw_sockets:1.

Now you have DHCP off, are you wanting it on or do you want a static IP?
 

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
I'd rather use a static IP, but I can live with setting a static DHCP lease to get the same result.
Pinging is not the main issue ; TCP / UDP connections are not working either from the jail. Just resolving domain names would be great.
 

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
(Sorry for double posting, I don't think I can edit my posts.)
Here is a demonstration of the connectivity issues:

Code:
root@freenas[~]# host google.com 10.12.0.1
Using domain server:
Name: 10.12.0.1
Address: 10.12.0.1#53
Aliases:

google.com has address 172.217.18.206
google.com has IPv6 address 2a00:1450:4007:805::200e
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
root@freenas[~]#
root@freenas[~]# iocage exec plexmediaserver host google.com 10.12.0.1
;; connection timed out; no servers could be reached
Command: host google.com 10.12.0.1 failed!
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
You need to confirm you can communicate with the gateway, so validate that first.

Have you defined a DNS server in the jails /etc/resolv.conf? Can be done from the GUI
 

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
I cannot communicate with the gateway with TCP, UDP nor ICMP.

Code:
root@freenas[~]# iocage get allow_raw_sockets plexmediaserver
1
root@freenas[~]# iocage exec plexmediaserver ping -c4 10.12.0.1
PING 10.12.0.1 (10.12.0.1): 56 data bytes

--- 10.12.0.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Command: ping -c4 10.12.0.1 failed!
root@freenas[~]# iocage exec plexmediaserver cat /etc/resolv.conf
nameserver 10.12.0.1
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
You need to try an get rid of that 0.0.0.0 alias from the jail epair0b interface, have you tried assigning an IP with DHCP enable, Im not really sure how you ended up with that.

The rest of your config looks fine to me, one other thing to try is: create a new fresh jail, create a bridge on the host called something like bridge100 with em0 as a member, then in your new jails interfaces use vnet0:bridge100.

Your bridge0 actually does look ok to me, but there are reports of odd behavior with it.

You could also just try NAT to see if that works to begin with...
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Esxi is a bit outside my wheelhouse but I do recall some posts recently about the promiscuous mode setting in the vswitch having something to do with jail connectivity. A search of the forum should turn up what you need.
 

executifs

Cadet
Joined
Dec 27, 2019
Messages
8
Wow Jailer, good call!
I have no idea why it worked in the past as I don't recall ever changing my vswitch settings, but allowing promiscuous mode immediately solved the issue. A million thanks to you!
 
Top