Plex stopped after 9.2.1.7 Upgrade

[CaCO2]

Hello,

First post here but have been using FreeNAS for a few years now without issue. Anyway I've got a simple setup, 6 drives configured for RaidZ2 with 1 dataset and 1 jail running the Plex plugin. Everything has been running smooth but last night I did an upgrade from 9.2.1.3 to 9.2.1.7 and immediately following the plex plugin stopped responding. By that I mean it stop responding externally via the web interface on port 32400. Internally via the FreeNAS gui I can still stop/start the jail or the plugin itself. From a different pc on the network I can ping it successfully. However if I do an NMAP scan it says it appears down. Restarted everything several times and even successfully upgraded the plex server via the FreeNAS gui with no effect. I can shell into the jail and see that everything is still there FWIW. Looking for ideas why this may have happened or if anyone can help me troubleshoot as I'm not as versed in BSD as I am Linux. Suggestions?

John....

 

[dlavigne]

Is the plex service running in the jail? Look for its port number in the output of sockstat -4. Also, is it showing as ON in the Plugins -> Installed tab?

 

[CaCO2]

Is the plex service running in the jail? Look for its port number in the output of sockstat -4. Also, is it showing as ON in the Plugins -> Installed tab?

Yes its running in the jail and it is ON in the Plugins -> Installed tab. Below is the output of sockstat. Still can't get a response from any other local pc to its IP and Port (192.168.1.176:32400).

Code:

root@plexmediaserver_1:/ # sockstat -4 -l                                     
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
plex     python     18970 10 tcp4   *:62599               *:*                 
plex     Plex Media 18969 52 tcp4   *:32400               *:*                 
plex     Plex Media 18969 53 tcp4   *:32401               *:*                 
plex     Plex Media 18969 70 udp4   *:32414               *:*                 
plex     Plex Media 18969 75 udp4   *:32410               *:*                 
plex     Plex Media 18969 76 udp4   *:32413               *:*                 
root     python2.7  11624 3  tcp4   192.168.1.176:12346   *:*                 
root     syslogd    11367 7  udp4   *:514                 *:*                 
root@plexmediaserver_1:/ #   

 

[dlavigne]

That is weird if local hosts can ping and the service is on.... Might be time to try a tcpdump or other packet sniffer.

 

[CaCO2]

That is weird if local hosts can ping and the service is on.... Might be time to try a tcpdump or other packet sniffer.

Ok this is where a get a little weak in the skill set. However I did run a quick tcpdump from the jail (192.168.1.176) while I tried connecting from my other pc (192.168.1.150). From what little I can tell I see a series of requests reaching the jail and Plex responding. Beyond that I don't know how to interpret it. So any help is appreciated. BTW, on the connecting pc I still get a time-out on the connection. BTW, still get same time-out on other devices as well so it not just the one pc. Thanks,

Code:

20:27:23.448540 IP 192.168.1.150.60848 > plexmediaserver_1.32400: Flags [R], seq 965707493, win 0, length 0                        
20:27:23.701508 IP 192.168.1.150.60849 > plexmediaserver_1.32400: Flags , seq 2081916964, win 14600, options [mss 1460,sackOK,TS val 324165572 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:23.701522 IP plexmediaserver_1.32400 > 192.168.1.150.60849: Flags [S.], seq 1182376941, ack 2081916965, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2026688363 ecr 324165572], length 0                                                              
20:27:23.701698 IP 192.168.1.150.60849 > plexmediaserver_1.32400: Flags [R], seq 2081916965, win 0, length 0                       
20:27:24.554883 IP 192.168.1.150.60850 > plexmediaserver_1.32400: Flags , seq 1812549621, win 14600, options [mss 1460,sackOK,TS val 324165828 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:24.554902 IP plexmediaserver_1.32400 > 192.168.1.150.60850: Flags [S.], seq 3421745509, ack 1812549622, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1800816112 ecr 324165828], length 0                                                              
20:27:24.555129 IP 192.168.1.150.60850 > plexmediaserver_1.32400: Flags [R], seq 1812549622, win 0, length 0                       
20:27:24.808227 IP 192.168.1.150.60851 > plexmediaserver_1.32400: Flags , seq 2634792487, win 14600, options [mss 1460,sackOK,TS val 324165904 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:24.808243 IP plexmediaserver_1.32400 > 192.168.1.150.60851: Flags [S.], seq 4160219745, ack 2634792488, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3494881646 ecr 324165904], length 0                                                              
20:27:24.808416 IP 192.168.1.150.60851 > plexmediaserver_1.32400: Flags [R], seq 2634792488, win 0, length 0                       
20:27:25.280944 IP li322-212.members.linode.com.https > plexmediaserver_1.32391: Flags [P.], seq 37:74, ack 1, win 252, options [nop,nop,TS val 282264034 ecr 95152754], length 37                                                                                     
20:27:25.380386 IP plexmediaserver_1.32391 > li322-212.members.linode.com.https: Flags [.], ack 74, win 1040, options [nop,nop,TS val 95162754 ecr 282264034], length 0                                                                                                
20:27:31.474847 IP 192.168.1.150.60848 > plexmediaserver_1.32400: Flags , seq 965707492, win 14600, options [mss 1460,sackOK,TS val 324167904 ecr 0,nop,wscale 7], length 0                                                                                         
20:27:31.474874 IP plexmediaserver_1.32400 > 192.168.1.150.60848: Flags [S.], seq 347235431, ack 965707493, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2106261690 ecr 324167904], length 0                                                                
20:27:31.475109 IP 192.168.1.150.60848 > plexmediaserver_1.32400: Flags [R], seq 965707493, win 0, length 0                        
20:27:31.714800 IP 192.168.1.150.60849 > plexmediaserver_1.32400: Flags , seq 2081916964, win 14600, options [mss 1460,sackOK,TS val 324167976 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:31.714812 IP plexmediaserver_1.32400 > 192.168.1.150.60849: Flags [S.], seq 1475030127, ack 2081916965, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3313373258 ecr 324167976], length 0                                                              
20:27:31.715044 IP 192.168.1.150.60849 > plexmediaserver_1.32400: Flags [R], seq 2081916965, win 0, length 0                       
20:27:32.568069 IP 192.168.1.150.60850 > plexmediaserver_1.32400: Flags , seq 1812549621, win 14600, options [mss 1460,sackOK,TS val 324168232 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:32.568088 IP plexmediaserver_1.32400 > 192.168.1.150.60850: Flags [S.], seq 2835493483, ack 1812549622, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1047634903 ecr 324168232], length 0                                                              
20:27:32.568323 IP 192.168.1.150.60850 > plexmediaserver_1.32400: Flags [R], seq 1812549622, win 0, length 0                       
20:27:32.834820 IP 192.168.1.150.60851 > plexmediaserver_1.32400: Flags , seq 2634792487, win 14600, options [mss 1460,sackOK,TS val 324168312 ecr 0,nop,wscale 7], length 0                                                                                        
20:27:32.834832 IP plexmediaserver_1.32400 > 192.168.1.150.60851: Flags [S.], seq 1332872427, ack 2634792488, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2111642641 ecr 324168312], length 0                                                              
20:27:32.835057 IP 192.168.1.150.60851 > plexmediaserver_1.32400: Flags [R], seq 2634792488, win 0, length 0                       
20:27:35.280876 IP li322-212.members.linode.com.https > plexmediaserver_1.32391: Flags [P.], seq 74:111, ack 1, win 252, options [nop,nop,TS val 282267033 ecr 95162754], length 37                                                                                    
20:27:35.380387 IP plexmediaserver_1.32391 > li322-212.members.linode.com.https: Flags [.], ack 111, win 1040, options [nop,nop,TS val 95172754 ecr 282267033], length 0  

 

[dlavigne]

For some reason, the client is resetting the connection. As an example, if you look at the 3-way handshake in lines 2-4, the client sends the SYN (S) to request a connection, the PMS sends the SYN ACK to say yes, I'm accepting connections, but the client then sends a RESET (R) instead of the ACK that is needed to complete the 3-way handshake. This means the connection never completes.

Has anything changed on the client side? Perhaps the client software or its configuration? Are there any firewall rules that may be in play on the client or in the network? An MTU issue? Something else you can think of?

 

[CaCO2]

Interesting. But the client is just Firefox and no changes there. I also just repeated the same connection and dump via a wifi android table (192.168.1.90) on the same LAN. Same results and same 3-way w/ RESET. Thinking there might be something in the ACK from plex that the clients don't like??? The only other thing that changed during that time is an upgrade I did to my pfsense firewall but I didn't change any rules only the OS upgrade. From the new dump below you can see that plex is talking with pfsense but I'm not sure what one has to do with the other related to this.

Code:

21:11:06.587407 IP 192.168.1.90.46650 > plexmediaserver_1.32400: Flags , seq 2544826114, win 14600, options [mss 1460,sackOK,TS val 1372568 ecr 0,nop,wscale 6], length 0                                                                                           
21:11:06.587420 IP plexmediaserver_1.32400 > 192.168.1.90.46650: Flags [S.], seq 3911716707, ack 2544826115, win 65535, options [mss1460,nop,wscale 6,sackOK,TS val 1020993913 ecr 1372568], length 0                                                                 
21:11:06.590513 IP 192.168.1.90.46650 > plexmediaserver_1.32400: Flags [R], seq 2544826115, win 0, length 0                        
21:11:06.833531 IP 192.168.1.43.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlecast._tcp.local. (40)                                  
21:11:06.833975 ARP, Request who-has 192.168.1.43 tell 192.168.1.76, length 46                                                     
21:11:07.206523 IP plexmediaserver_1.10698 > pfsense.localhost.localdomain.domain: 5448+ PTR? 90.1.168.192.in-addr.arpa. (43)      
21:11:07.214224 IP pfsense.localhost.localdomain.domain > plexmediaserver_1.10698: 5448 NXDomain 0/0/0 (43)                        
21:11:07.214385 IP plexmediaserver_1.52210 > pfsense.localhost.localdomain.domain: 5449+ PTR? 251.0.0.224.in-addr.arpa. (42)       
21:11:07.219640 IP pfsense.localhost.localdomain.domain > plexmediaserver_1.52210: 5449 NXDomain 0/1/0 (99)                        
21:11:07.219690 IP plexmediaserver_1.27217 > pfsense.localhost.localdomain.domain: 5450+ PTR? 43.1.168.192.in-addr.arpa. (43)      
21:11:07.224345 IP pfsense.localhost.localdomain.domain > plexmediaserver_1.27217: 5450 NXDomain 0/0/0 (43)                        
21:11:07.224409 IP plexmediaserver_1.61986 > pfsense.localhost.localdomain.domain: 5451+ PTR? 76.1.168.192.in-addr.arpa. (43)      
21:11:07.229676 IP pfsense.localhost.localdomain.domain > plexmediaserver_1.61986: 5451 NXDomain 0/0/0 (43)                        
21:11:11.361153 ARP, Request who-has plexmediaserver_1 tell 192.168.1.90, length 50                                                
21:11:11.361162 ARP, Reply plexmediaserver_1 is-at 02:67:95:00:09:0b (oui Unknown), length 28                  

 

[dlavigne]

Did you ever figure this one out?

 

[CaCO2]

Unfortunately not. So as time is my enemy I wound up just creating a 2nd jail with a new Plex build and rebuilt the cover art and other meta data. Still have the original jail for now in case I have an ah-ha moment. BTW, thanks for all the help trying to dig into this one!

John....