Jails keep having IP instability

[francisaugusto]

Hi,

I use FreeNAS since version 11.0, and since then I have trouble with jails. I am now on the latest version (11.3-U3.2) Some of my jails - especially, but not only - Nextcloud, will have a problem with IP addresses. I choose DHCP, I set it to static, I restart the jail, and nothing - it won't resolve DNS, it won't ping anything outside of the FreeNAS. It has happened always. Lately it has been more stable, but yesterday it happened again.

I searched this forums, and I've seen many people have similar problems, though there's no straightforward fix for these.

Luckily for me a FreeNAS restart solved the problem yesterday, but are Jails THAT unreliable? I am seriously thinking about moving my nextcloud installation to an Ubuntu VM. I already did move it from a plugin to a jail (ie., I created a jail and installed NC instead of using the plugin, because somehow it would always mess-up after an update).

Any thoughts on these issues, or is it just me?

 

[Samuel Tai]

On bare metal, my jails are incredibly stable. You're running FreeNAS as a VM, but haven't provided any info on how you're passing disks into FreeNAS or providing networking to the VM.

 

[francisaugusto]

On bare metal, my jails are incredibly stable. You're running FreeNAS as a VM, but haven't provided any info on how you're passing disks into FreeNAS or providing networking to the VM.

I forgot to mention: my VM setup was migrated from bare metal last year, and the problem was the same. Disks are passed via PCI passthrough, and the network is provided via DHCP. The port group and vswitch are in promiscuous mode.

 

[Samuel Tai]

What network driver are you using for FreeNAS? The VMXNET3 or the emulated E1000?

 

[Patrick M. Hausen]

If you set the IP address statically for the jail you will also have to manually set the default-gateway and the DNS server. This is the most common cause of your problem.

 

[Samuel Tai]

Also, please describe the switch you're using for the ESX host.

 

[francisaugusto]

What network driver are you using for FreeNAS? The VMXNET3 or the emulated E1000?

I use VMXNET3.

Also, please describe the switch you're using for the ESX host.

It's a managed Unifi switch - you mean the physical switch the host is connected to?

 

[Samuel Tai]

Yes, please describe the switchport configuration of the Unifi switch, and also the vSwitch in the host.

 

[francisaugusto]

If you set the IP address statically for the jail you will also have to manually set the default-gateway and the DNS server. This is the most common cause of your problem.

I don't see a field for that, actually, but some of my jails have static ip and they work fine. And if that was the problem, why only restarting the FreeNAS would solve it?

Don't get me wrong, it could be misconfiguration, but my point is that this kind of instability where things work well, stop working without any apparent cause and then work again after a restart of the whole system is annoying.

 

[francisaugusto]

Yes, please describe the switchport configuration of the Unifi switch, and also the vSwitch in the host.

No VLANs there, simple port group on the vswitch. Pretty default, except for the promiscuous mode.

 

[Samuel Tai]

Can you provide the output of iocage get all <name of NC jail>?

 

[Patrick M. Hausen]

@francisaugusto ok, I did not read "I choose DHCP, I set it to static, I restart the jail, and nothing - it won't resolve DNS, it won't ping anything outside of the FreeNAS." as "this works for a while then unexpectedly stops". I read it as "this does not work at all."

Please be assured that jails are a rock-solid technology if there ever was one. I run a full datacenter of ~100 hosts with ~1000 jails on them ...

@Samuel Tai is probably on a good track with you.,

 

[francisaugusto]

Can you provide the output of iocage get all <name of NC jail>?

Sure:

Code:

# iocage get all new-nextcloud
CONFIG_VERSION:26
allow_chflags:0
allow_mlock:0
allow_mount:1
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:1
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:1
bpf:1
children_max:0
cloned_release:11.3-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.1.1
defaultrouter6:auto
depends:none
devfs_ruleset:8
dhcp:1
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:med-lo
host_hostname:new-nextcloud
host_hostuuid:new-nextcloud
host_time:1
hostid:e69c29f4-dc6f-11e8-9c3d-6c626d871dba
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.1.55/24
ip4_saddrsel:1
ip6:new
ip6_addr:vnet0|accept_rtadv
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/new-nextcloud/data
jail_zfs_mountpoint:none
last_started:2020-07-08 18:37:50
localhost_ip:none
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:11.3-RELEASE-p10
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:02ff607fbee9 02ff607fbeea
vnet1_mac:none
vnet2_mac:none

 

[Samuel Tai]

Here's a discrepancy that may account for your instability:

Code:

allow_mount_devfs:0
mount_devfs:1

Try shutting down the jail, and running iocage set allow_mount_devfs=1 new-nextcloud. Then restart the jail.

Also, when the jail is running, please provide the output of ifconfig -a outside the jail. There should be something like this:

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether xx:xx:xx:xx:xx
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vmx0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: new-nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:7f:be:e9 
        hwaddr xx:xx:xx:xx:xx
        inet6 xxxx::ff:60ff:7fbe:e9%vnet0.1 prefixlen 64 scopeid 0x6
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

 

[Fredda]

The output of iocage exec <jailname> netstat -r might be of interest.

 

[Samuel Tai]

The output of iocage exec <jailname> netstat -r might be of interest.

The default router is in the iocage get all new-nextcloud output:

Code:

defaultrouter:192.168.1.1
defaultrouter6:auto
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.1.55/24
ip4_saddrsel:1
ip6:new
ip6_addr:vnet0|accept_rtadv
ip6_saddrsel:1

 

[Fredda]

OK. Missed that. Never mind...

 

[francisaugusto]

Here's a discrepancy that may account for your instability:

Code:

allow_mount_devfs:0
mount_devfs:1

Try shutting down the jail, and running iocage set allow_mount_devfs=1 new-nextcloud. Then restart the jail.

Also, when the jail is running, please provide the output of ifconfig -a outside the jail. There should be something like this:

I got this:

Code:

        bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ec:e5:de:82:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: new-nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:7f:be:e9
        hwaddr 02:e5:50:00:09:0a
        inet6 fe80::ff:60ff:fe7f:bee9%vnet0.6 prefixlen 64 scopeid 0x9
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

Things are working, but could you explain what does that change you said mean?

 

[Samuel Tai]

You asked the jail to mount /dev, but didn't give it permission to mount /dev. Also, your bridge configuration seems incomplete: it's not bound to one of the parent FreeNAS vmx interfaces as in my example.

 

[francisaugusto]

You asked the jail to mount /dev, but didn't give it permission to mount /dev. Also, your bridge configuration seems incomplete: it's not bound to one of the parent FreeNAS vmx interfaces as in my example.

Sorry, my bad. it is bound to vmx - I didn't paste it the first time:

Code:

        member: vmx0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000