Can't reach any jails's web UI on 9.10.2-U3

[chrisada]

Hi all,

My box has been running for abour 1.5 years. I upgraded from 9.10.2-U2 to U3 on 2017-05-02, and the jails were working up until last night (2017-05-09). I started to notice something was wrong when my Plex client couldn't connect to the server. After a quick check, I saw that all my plugins are OFF and cannot be switched ON. These include CouchPotato, Emby, Syncthing. The Plex server is actually a custom jail running Plexpass version of the server.

I couldn't reach any of the web UI, both for the ones on standard jails nor plugin jails. Chrome says

Code:

This site can’t be reached

192.168.100.23 refused to connect.

ERR_CONNECTION_REFUSED

But they do show up on my router's client list and I can ping them from my Windows laptop.

Code:

Pinging 192.168.100.23 with 32 bytes of data:
Reply from 192.168.100.23: bytes=32 time=1ms TTL=64
Reply from 192.168.100.23: bytes=32 time=1ms TTL=64
Reply from 192.168.100.23: bytes=32 time=1ms TTL=64
Reply from 192.168.100.23: bytes=32 time=2ms TTL=64

Ping statistics for 192.168.100.23:
	Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
	Minimum = 1ms, Maximum = 2ms, Average = 1ms

In reverse, I can ping my laptop from FreeNAS shell. Trying to ping my laptop from within the jails, I got "socket: Operation is not permitted". (Not sure if this was working before the problem)

Each jail is given a static IPv4 from within FreeNAS, with VIMAGE checked.

A couple of things I tried that didn't help:

• turn off VIMAGE and selecting the NIC manually.
• add IPv4 default gateway
• turn on NAT

My hardwares are:

• Xeon E3-1231 v3
• ASRock E3C226D2I
• Kingston ECC 8GB/1600 X 2
• Boot: SSD Sandisk Plus 120GB
• Storage: 5 X WD Red 3TB

I'd appreciate very much any tips on what I could try to resolve this. The toddler is without her cartoons and the withdrawal is severe.

 

[SweetAndLow]

In the jail can you do service plexmediaserver onestart?

Sent from my Nexus 5X using Tapatalk

 

[chrisada]

In the jail can you do service plexmediaserver onestart?

This is really strange! I assume this is a network issue and the above command should work. But then I get:

Code:

plexmediaserver does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d)

I want to try to upgrade the package, thinking it might help, but then:

Code:

# pkg upgrade
pkg: /var/db/pkg wrong user or group ownership (expected 0/0 versus actual 666/666)

That can't be normal.

 

[SweetAndLow]

I must have gotten the name of the service Wong. What is the name of the service? cat /etc/rc.d.conf should give you a hint.

Did you change any permissions recently? I suspect you broke all your jails by modifying a parent dataset.

Sent from my Nexus 5X using Tapatalk

 

[chrisada]

I must have gotten the name of the service Wong. What is the name of the service? cat /etc/rc.d.conf should give you a hint

In my case, it is plexmediaserver_plexpass

Code:

# service plexmediaserver_plexpass onestart
Starting plexmediaserver_plexpass.
su: pam_start: system error
/usr/local/etc/rc.d/plexmediaserver_plexpass: WARNING: failed to start plexmediaserver_plexpass

I want to note that this is not a plugin. But the other 5 plugins also stopped working with the same behavior (can be pinged, cannot access web UI) Plugins are Sonarr, Sabnzbd, Emby, Syncthing, Couchpotato

 

[SweetAndLow]

Exactly! you changed the permissions of your root dataset and hit recursive or something. You have to rebuild all your plugins now. You can't change the permissions of your jails, it breaks everything. Kind of like just doing a chmod -R 777 / on a normal system, everything will break.

 

[chrisada]

It does seem to be something system-wide, but I can't understand how it happened. It was working Monday and Tuesday, and the only thing I know I did was turning on webdav service and creating a webdav share. (not on the jails dataset)

Is there a way to be more certain about the nature of the problem? It would be a shame to have to rebuild all the jails, but not impossible. I just want to make sure that will really solve the problem. (wouldn't want to go through all that only to find out the problem lies elsewhere)

Sent from my SM-G935F using Tapatalk

 

[SweetAndLow]

I'm 100% positive you have to rebuild your jails. If you can show me the permissions on your jail dataset or what the webdav settings look like that would be helpful. it was probably the webdav thing because it sets the owner of everything to the webdav user. What dataset are you using for webdav and does it live above the jails?

 

[chrisada]

I'm 100% positive you have to rebuild your jails. If you can show me the permissions on your jail dataset or what the webdav settings look like that would be helpful. it was probably the webdav thing because it sets the owner of everything to the webdav user. What dataset are you using for webdav and does it live above the jails?

Gah. You're right about webdav changing owner of everything!!! I was only trying it out of curiosity and had since removed the share and disable the service, but now everything is owned by webdav user. Some warning popup would have been nice!

This affect my "data" datasets too, but those are probably easier to fix by recursively changing owners to what is supposed to be.

For jails, sigh.. wish I had learn how to use snapshots.

Sent from my SM-G935F using Tapatalk

 

[danb35]

it was probably the webdav thing because it sets the owner of everything to the webdav user

Does WebDAV do that for "everything" everything, or just "everything in/under the WebDAV directory"? If the former, that sounds like a pretty big bug.

 

[SweetAndLow]

Under the path

Sent from my Nexus 5X using Tapatalk

 

[SweetAndLow]

Gah. You're right about webdav changing owner of everything!!! I was only trying it out of curiosity and had since removed the share and disable the service, but now everything is owned by webdav user. Some warning popup would have been nice!

This affect my "data" datasets too, but those are probably easier to fix by recursively changing owners to what is supposed to be.

For jails, sigh.. wish I had learn how to use snapshots.

Sent from my SM-G935F using Tapatalk

It does tell you in multiple places, it tells you in the documentation and it also tells you when creating a webdav share in the GUI, there is even a checkbox for this and you can remove it. Pretty tough to give you anymore warning.

 

[chrisada]

It does tell you in multiple places, it tells you in the documentation and it also tells you when creating a webdav share in the GUI, there is even a checkbox for this and you can remove it. Pretty tough to give you anymore warning.

You're right. I added the share on my phone and missed the warning. It didn't occur to me that creating a share (although a type I'm not familiar with) could potentially change things irreversibly. We live & we learn.

Sent from my SM-G935F using Tapatalk

 

[chrisada]

Does WebDAV do that for "everything" everything, or just "everything in/under the WebDAV directory"? If the former, that sounds like a pretty big bug.

It changed owner/group of everything under the path chosen for the share. In this incident, I selected my main dataset, so everything got changed. (including jails dataset but not iohyve dataset, somehow)