Changed routers and jails went down.

[MomSpaghetti]

I'm running FreeNAS 11.1 and have several jails with Plex, Sickrage, Transmission, and others running. I configured pfsense today and was setting up firewall rules to remotely access plex. Suddenly I could no longer access plex from <jailIP>:32400 like I always have before. I played around with various pfsense settings including NAT and upnp, but I never changed a single setting in FreeNAS, or any of my jails. I gave pfsense the IP of my old router, so default gateways never changed.

I switched back to my old router, and have the exact same problem, I can't reach Plex or my most of my other jails, they refuse to connect. For some reason sickrage still works, but nothing else. When I jexec into my jails, the sickrage jail is the only one that can ping the gateway and other devices on my network. Pinging from plex jail gives me a

Code:

ping: ssend socket: Operation not permitted.

I rolled my plex jail back to a snapshot form yesterday, and have the same result. I've triple checked my dns and masks. I've restarted the jails and FreeNAS several times during this whole process. ifconfig in each of the jails shows the correct info. I don't know where to go from here.

EDIT** pinging 127.0.0.1 from the jails that are down fails as well.
EDIT2** I changed one of the jails to an iocage jail, and have the exact same problem. It won't ping local host.

 

[Jailer]

Does the contents of /etc/resolv.conf in the jails match your default gateway listed in your FreeNAS GUI?

 

[MomSpaghetti]

Does the contents of /etc/resolv.conf in the jails match your default gateway listed in your FreeNAS GUI?

Yes, each jails /etc/resolv.conf matches the FreeNAS GUI. I just copied the FreeNAS's /etc/resolv.conf file to one of my jails as a test, and nothing changed. They were the same to begin with, but I thought I would try since FreeNAS itself was working.

 

[Jailer]

Have you tried rebooting your server? I had this happen once when I had to change my pfsense hardware. Everything worked but I couldn't connect out from my jails. A reboot solved it.

 

[Elliot Dierksen]

I gave pfsense the IP of my old router, so default gateways never changed.

The IP address might not have changed, but the ARP entry associated with that IP definitely changed. It is likely that some of the jails/hosts had that old ARP entry cached, and it hadn't been cleared yet.

Have you tried rebooting your server?

That would certainly clear the ARP cache. It can be done other ways depending on the platform, but a reboot certainly classifies as 'take off and nuke the place from orbit, only way to be sure'. :)

 

[MomSpaghetti]

Have you tried rebooting your server? I had this happen once when I had to change my pfsense hardware. Everything worked but I couldn't connect out from my jails. A reboot solved it.

Yea I've reboot several times.

 

[MomSpaghetti]

The IP address might not have changed, but the ARP entry associated with that IP definitely changed. It is likely that some of the jails/hosts had that old ARP entry cached, and it hadn't been cleared yet.

That would certainly clear the ARP cache. It can be done other ways depending on the platform, but a reboot certainly classifies as 'take off and nuke the place from orbit, only way to be sure'. :)

I forgot about ARP, but the server and router have been restarted several times since the problem started, so something else is going on.

 

[Elliot Dierksen]

I forgot about ARP, but the server and router have been restarted several times since the problem started, so something else is going on.

Something else might have cached the old ARP entry, and be responding to it for other devices on your network. ifconfig -a on Unix or ipconfig /all on windows will show you the MAC address of your interfaces. arp -a on both platforms will show you the ARP table on a given host. It is worth looking to see if that is the problem.

 

[MomSpaghetti]

Something else might have cached the old ARP entry, and be responding to it for other devices on your network. ifconfig -a on Unix or ipconfig /all on windows will show you the MAC address of your interfaces. arp -a on both platforms will show you the ARP table on a given host. It is worth looking to see if that is the problem.

arp -a from my PC (Linux Mint) shows the correct IP and MAC for the plex jail. arp -a from the plex jail shows two entries, itslef, and the gateway with the correct IP and correct MAC of old router (the one currently in use).

 

[Elliot Dierksen]

arp -a -d will dynamically delete all ARP entries without a reboot. If you do that and the old ARP entry comes back, something on your network has the old entry cached and is doing proxy-arp (or something close to that). Normally only the device associated with the IP address being resolved should respond, but the ARP request is a broadcast. That means other things could see it, and respond with an incorrect cached entry.

 

[MomSpaghetti]

arp -a -d will dynamically delete all ARP entries without a reboot. If you do that and the old ARP entry comes back, something on your network has the old entry cached and is doing proxy-arp (or something close to that). Normally only the device associated with the IP address being resolved should respond, but the ARP request is a broadcast. That means other things could see it, and respond with an incorrect cached entry.

That makes sense, but unfortunately did not work. I performed arp -a -d on FreeNAS, and in each jail. I think whatever is causing the ping: ssend socket: Operation not permitted error when I ping is the main problem.

 

[Jurgen Segaert]

If you want to use ping and alike from a jail, be sure that the allow_raw_sockets parameter is set.

Code:

iocage set allow_raw_sockets=1 <jail>