Need help debugging unresponsive freenas

[ovizii]

Here's what happened:

- freenas running on HP N54L with 5 x 1TB HDs encrypted and 8GB RAM (NON ECC) - ECC is ordered so this had to do for now
- system had been running for about 2 months, I did a reinstall in-between as I added an additional HD to the mix
- boots off 2 USB sticks that are mirrored

Yesterday, all working fine, I left for a barbeque, accessed the freenas remotely from my Android phone via a proxy and accessed a cifs share to check a picture.

Came home 4h later accessed the GUI which was open in my browser and clicked: check for update got a reply of cannot resolve host !? Tried changing DNS, etc, nothing helped. From my MacBook all good accessing the internet. I was still logged in via SSH and even from the cl I couldn't even ping anything outside my network.

I realized some cl commands showed this kind of error:

Code:

isc_socket_create: not enough free resources
can't create socket - .... no buffer space available

I checked memory and all was fine ,the freenas was basically idling...

I then decided to try a reboot. After the reboot the freenas wouldn't boot but show these errors when connecting a screen to it so I selected the second to last boot environment and it booted (while doing this I was connected via a network cable to my MacBook) - then moved the freenas to its proper place where I have no access to a screen and its booting but not getting an IP so I assume it's the same problem as above.

The only suspicion I have is the last change I did before leaving the house but I'm unsure how to reverse this from the command line:

- I had found errors in transmission's logs and after googling it I found a solution in these forums here: https://forums.freenas.org/index.ph...-receive-buffer-please-help.15203/#post-74611

The errors were something like this:

Code:

UDP failed to set receive buffer: No buffer space available (tr-udp.c:58)
UDP failed to set receive buffer: requested 4194304, got 42080 (tr-udp.c77)

so accordingly to that thread I used the GUI to set these tunables:

Code:

kern.ipc.maxsockbuf=5242880
net.inet.udp.recvspace=5242880

both set to sysctl

Could this be the cause of my issues and how to revert this from the command line?

 

[Robert Trevellyan]

Could this be the cause of my issues and how to revert this from the command line?

I know this won't be super helpful, but here goes. Did you backup the config before you made this change? If so, you could reset to factory defaults from the console, then upload that previous config.

 

[ovizii]

no,no, every advice is welcome.
I have a slightly older backup but haven'T changed anything significant since so I'll give that a try and then report back here.
will take a bit as I have to move the NAS back to the location with the screen :-/

 

[ovizii]

I know this won't be super helpful, but here goes. Did you backup the config before you made this change? If so, you could reset to factory defaults from the console, then upload that previous config.

before I do that, do you know where to find the current config via cli so I can save it just in case?

 

[Bidule0hm]

It's here: /data/freenas-v1.db ;)

 

[Robert Trevellyan]

I have a slightly older backup but haven'T changed anything significant since so I'll give that a try and then report back here.

At least then you'll be closer to knowing whether those tunables are the issue.

 

[cyberjock]

You realize that thread is from 2013 and applies to a version of FreeNAS that is nothing like what we use now. Unfortunately, you made the buffer smaller. I'm not kidding. The default is 16MB, and you shrunk it to about 5 MB. That is not the droid you are looking for. It would be a good idea to see what the default values are before you change them. ;)

Frankly, I think the UDP errors are a red herring to the bigger problem. What that problem is, I don't know. My first guess would be a boot device that is not really compatible with FreeNAS or is corrupted. Also, unless you are using NFS with UDP support enabled I can't imagine you getting enough UDP traffic for things to matter.

I don't know what else you use the system for, but if you are using things like Plex or other jails/plugins that are RAM intensive, the minimum RAM is not 8GB, its more like 12GB+ because the 8GB minimum excludes any jails or plugins you are using. I also have no clue how much more RAM is required for encrypted zpools. Generally I don't recommend zpools that are encrypted unless required by law because of other problems that can develop. This may or may not fall into that "other problems that can develop" because of how little RAM you have. I don't know if I've seen anyone build a system with the minimum RAM and then chose to use encryption. There may be problems with that little RAM.

 

[fta]

You realize that thread is from 2013 and applies to a version of FreeNAS that is nothing like what we use now. Unfortunately, you made the buffer smaller. I'm not kidding. The default is 16MB, and you shrunk it to about 5 MB. That is not the droid you are looking for. It would be a good idea to see what the default values are before you change them. ;)

Code:

root@freenas:~# sysctl kern.ipc.maxsockbuf
kern.ipc.maxsockbuf: 2097152
root@freenas:~# sysctl net.inet.udp.recvspace
net.inet.udp.recvspace: 42080
root@freenas:~# cat /etc/version
FreeNAS-9.3-STABLE-201506042008

 

[ovizii]

@cyberjock @fta
If I read fta's reply correctly, I have NOT made the buffers smaller? His reply seems to indicate I have done the right thing?
Anyway, as soon as I can move the NAS I'll test this, replace my old backup and let you guys know what's what.

- not using NFS
- there is plex, syncthing, couchpotato, transmission and sickrage installed BUT NONE OF THEM was actively doing anything. as in I haven't downloaded anything in ages, and was not watching any movies. As I said, I was at a barbecue and the RAM and CPU usage looked just fine.
- 16GB ECC is on its way
- Before changing the values myself I had autotune running it had set kern.ipc.maxsockbuf: 2097152 on its own


BTW: I had quite a few problems getting GRUB to boot my second to last config but now its up and running and I'm pretty sure it was those 2 sysctl variables that messed up. I'll just not do that again and see if everything will keep working, other than that I'm open to any suggestions.
I'll also look at any error logs I can find to see if we can figure out what went wrong here.

 

[Robert Trevellyan]

BTW: I had quite a few problems getting GRUB to boot my second to last config but now its up and running and I'm pretty sure it was those 2 sysctl variables that messed up.

Good to know.