Jail loses visibility of recursive mount

[johnjaylward]

I'm running a plex plugin jail and my mounts into it look something like this:

Code:

$ cat /mnt/jails/.plexmediaserver_1.meta/fstab
/mnt/storage/multimedia/Pictures /mnt/jails/plexmediaserver_1//mnt/multimedia/Pictures nullfs ro 0 0
/mnt/storage/multimedia/Video/Anime_Movies /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/Anime_Movies nullfs ro 0 0
/mnt/storage/multimedia/Video/TV_Shows /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/TV_Shows nullfs ro 0 0
/mnt/storage/multimedia/Video/Movies /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/Movies nullfs ro 0 0
/mnt/storage/multimedia/Video/HomeVideos /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/HomeVideos nullfs ro 0 0
/mnt/storage/multimedia/Video/Anime /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/Anime nullfs ro 0 0
/mnt/storage/multimedia/Video /mnt/jails/plexmediaserver_1//mnt/multimedia/Video nullfs ro 0 0
/mnt/storage/multimedia/Music /mnt/jails/plexmediaserver_1//mnt/multimedia/Music nullfs ro 0 0
/mnt/storage/multimedia /mnt/jails/plexmediaserver_1//mnt/multimedia nullfs ro 0 0

As you can see I have recursive datasets mounted into the jail (all configured through the UI using the "Add Storage" option on the jail.)

When the system boots the first time, the jail runs fine, and all the data can be seen. However, after a while (I haven't timed it) all the mounts under /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/ can not be seen from within the jail. They still show as mounted from outside the jail using the mount command, and I can navigate to /mnt/jails/plexmediaserver_1//mnt/multimedia/Video/ in the shell and see the top level directories, but those directories are all empty.

If I stop and start the jail, the datasets are still not visible. It doesn't work again until I restart the entire box. If I try to unmount and remount the datasets I appear to get a kernel panic and have to restart the box using either IMPI or physically going to the box and resetting it.

Any ideas as to what could be going wrong?

 

[SweetAndLow]

Hardware specs?

 

[johnjaylward]

8x WD Red WD40EFRX 4TB IntelliPower 64MB cache (for main storage)
1x LSI 9300-4i kit PCI-Express 3.0 SATA / SAS 4-Port SAS3 12Gb/s HBA
2x SAMSUNG 840 EVO MZ-7TE250BW 2.5" 250GB SATA (for jails)
1x usb thumb drive for OS
SUPERMICRO SYS-6027R-TRF 2U Rackmount Server
2x Intel Xeon E5-2603 Sandy Bridge-EP 1.8GHz 10MB L3 Cache LGA 2011 80W Server Processor
Wintec Server Series 64GB (4 x 16GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1600

The motherboard has duel intel NICs which I configured with LACP in Freenas and on my managed switch (NETGEAR ProSafe 24-port Gigabit Smart Switch GS728TP)

FreeNAS-9.3-STABLE-201412312006

Running Plugins:
Crashplan (although it's not configured)
Plex (the jail I'm seeing the problem in)
Transmission (will run for a while then the service will stop for some reason)

Running Services:
AFP
LLDP
NFS
S.M.A.R.T.
SSH
UPS

Custom Jails:
Standard jail (from 9.2) running mysql

 

[dlavigne]

That is weird. Anything related in /var/log/messages?

 

[johnjaylward]

I always get these every couple of minutes:

Code:

Jan  7 18:39:16 freenas ntpd[2850]: bind() fd 29, family AF_INET6, port 123, scope 0, addr fd00::ec4:7aff:fe14:6276, mcast=0 flags=0x11 fails: Can't assign requested address
Jan  7 18:39:16 freenas ntpd[2850]: unable to create socket on lagg0 (3669) for fd00::ec4:7aff:fe14:6276#123

I've also been getting these messages the last couple of days:

Code:

Jan  7 19:36:06 freenas kernel: ifa_del_loopback_route: deletion failed
Jan  7 19:36:06 freenas Freed UMA keg (udp_inpcb) was not empty (70 items).  Lost 7 pages of memory.
Jan  7 19:36:06 freenas Freed UMA keg (udpcb) was not empty (1008 items).  Lost 6 pages of memory.
Jan  7 19:36:06 freenas Freed UMA keg (tcp_inpcb) was not empty (60 items).  Lost 6 pages of memory.
Jan  7 19:36:06 freenas Freed UMA keg (tcpcb) was not empty (24 items).  Lost 6 pages of memory.
Jan  7 19:36:06 freenas Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.

and

Code:

Jan  7 19:16:55 freenas manage.py: [freeadmin.navtree:599] An error occurred while unserializing from http://192.168.55.70/plugins/plexmediaserver/3/_s/treemenu: No JSON object could be decoded

the first set of errors I found a related bug in the tracker that is closed as "won't fix"

the second set appears to be related to stopping jails

the third error seems to be coming from the Freenas UI and I've seen that since 9.2 and it doesn't seem to cause any issues.

Other than these messages, I only see the autosnapshot.py being executed.

 

[johnjaylward]

So, I deleted an "Own Cloud" plugin around the time I made the initial post, and so far the problem hasn't cropped up again. No idea if they were related or not, but I'm going to mark this as resolved for now, and if I see something similar, I'll try posting on the freebsd board instead.

 

[diskdiddler]

I don't use Own Cloud and I'm having what appears to be similar problems?

https://forums.freenas.org/index.ph...ping-for-jail-is-forgetting-mountpoint.27720/
https://forums.freenas.org/index.php?threads/security-run-output-explanation.26668/#post-178795

> Freed UMA keg (udp_inpcb) was not empty (80 items). Lost 8 pages of memory.
> Freed UMA keg (udpcb) was not empty (1008 items). Lost 6 pages of memory.
> Freed UMA keg (tcpreass) was not empty (1092 items). Lost 13 pages of memory.
> Freed UMA keg (tcptw) was not empty (400 items). Lost 8 pages of memory.
> Freed UMA keg (tcp_inpcb) was not empty (120 items). Lost 12 pages of memory.
> Freed UMA keg (tcpcb) was not empty (48 items). Lost 12 pages of memory.

 

[johnjaylward]

I still haven't found a resolution for this, but the "Lost pages of memory" message is apparently related to a problem with an Intel NIC driver. It should be unrelated to the mount visibility issue we are seeing.

I did start a new thread on the FreeBSD boards, but haven't had time to further debug the issue:
https://forums.freebsd.org/threads/jail-loses-visibility-of-recursive-null-mounts.50244/#post-281156

 

[blackmesa]

I have the same errors, when I start my btsync/owncloud jail, e.g. If it is related to Intel NIC drivers, how should I resolve this? I have a standard HP Microserver Gen8 G1610T.

 

[johnjaylward]

For the memory pages see these:
https://forums.freenas.org/index.ph...tput-problem-with-lost-pages-of-memory.22549/
https://bugs.freenas.org/issues/4058

They seem to indicate that it's not really a problem most of the time. If you want to try and fix it, you should use either of the other links, not this thread.

 

[diskdiddler]

Just to clarify, I'm using a HP Microserver N54L not the G8, the G7 series.
9.2.xx was fine on the machine - but these dropping mount points will be a problem if I need to reboot regularly.

 

[diskdiddler]

I got this mail this morning.

#7265 Bug Normal WebDAV cant stop
#7787 Bug Normal Create tooltip for Volume/Dataset field of Add Replication Task
#7894 Bug Normal Problem in webdav services script
#7998 Feature High Add IPMI IP to debug info
#8016 Bug Normal Jails; Storage not getting unmounted when removed through the UI
#8055 Bug Normal error adding new jail
#8062 Bug Low Minor (but confusing) error in Alert instructions on how to update LSI adapter firmware
#8064 Bug Normal Incorrect sas2flash instructions
#8066 Bug Normal Expunge istgt from my sight.
#8073 Bug Normal FTP Service
#8078 Bug Normal Disable Sendfile
#8098 Bug High VMware snapshot name to long

Could this be a fix?

 

[diskdiddler]

Rebooted - jails are still losing their mountpoint.
https://forums.freenas.org/index.ph...ping-for-jail-is-forgetting-mountpoint.27720/