Issues with this error - freenas kernel: sonewconn: pcb 0xfffff801c8f631d0: Listen queue overflow

[Noctrin]

Feb 27 17:07:09 freenas kernel: sonewconn: pcb 0xfffff801c8f631d0: Listen queue overflow: 151 already in queue awaiting acceptance (12 occurrences)
Feb 27 17:08:29 freenas kernel: sonewconn: pcb 0xfffff801c8f631d0: Listen queue overflow: 151 already in queue awaiting acceptance (2 occurrences)
Feb 27 17:09:58 freenas kernel: sonewconn: pcb 0xfffff801c8f631d0: Listen queue overflow: 151 already in queue awaiting acceptance (3 occurrences)
...

I've been having this for a while since 11.1 I think. It will go away after a server restart and be fine for a few weeks. I have tried looking at incoming connections and i don't see anything out of the ordinary, i also don't fully understand how to properly debug this. I googled furiously and followed a number of threads on similar issues, but i cannot get to the bottom of it. Here's hoping someone can help me with this.

Hardware:
Dell R510
2 x 8 core Intel
32GB Ram
1 Nic (onboard) connected to network (2nd port available)
1 VM running ubuntu and zoneminder, nextcloud and pi-hole -- other than nextcloud nothing is open to outside.
4 Jails: 1 for plex (open) 1 for radar, sonarr..etc, 1 jail for grafana and a few other monitoring tools and 1 jail for deluge

FreeNas 11.2 (Stable)

Probably running a few other things as well.

I can't say i see any errors or any abnormal behavior when these messages start and i can't seem to find a trigger for them. They just seem to randomly start and will not go away, even if i start turning off the VM or jails. After a reboot all is well for a few weeks.

Any help appreciated, thank you!

 

[Chris Moore]

32GB Ram

How much swap is getting used?

With all that going on, I bet you need more memory.

 

[Noctrin]

How much swap is getting used?

With all that going on, I bet you need more memory.

No swap usage, i only gave the VM 3 GB of RAM and have 6TB of storage

 

[ThomasDK81]

I am getting the same errors as mentioned here: https://www.ixsystems.com/community/threads/what-looks-like-a-problem.72140/

Ps. No swap being used either.

Did you resolve this?

 

[Noctrin]

I am getting the same errors as mentioned here: https://www.ixsystems.com/community/threads/what-looks-like-a-problem.72140/

Ps. No swap being used either.

Did you resolve this?

nope.. but, it doesnt seem to cause any issues. I just reboot the server once a month when it starts popping up.

 

[sretalla]

I get that error on a system with 128GB of RAM.

I get it fairly regularly, but can't put it together with anything that's an actual problem.

I tried in the past in another thread to work on tunables to increase the queue length, but even massive increases there didn't make a difference at all.

I stopped worrying about it.

 

[Noctrin]

I get that error on a system with 128GB of RAM.

I get it fairly regularly, but can't put it together with anything that's an actual problem.

I tried in the past in another thread to work on tunables to increase the queue length, but even massive increases there didn't make a difference at all.

I stopped worrying about it.

I'm upgrading my Dell R510 with a thinkserver RD450. Once i do, if the error persists, I can at least say with decent confidence the issue is software related.

 

[sretalla]

I'm upgrading my Dell R510 with a thinkserver RD450. Once i do, if the error persists, I can at least say with decent confidence the issue is software related.

I already saw the problem on a completely different mobo and NIC combination, so already confirm it's something to do with software.

 

[sretalla]

Oops, I posted a response to the wrong thread here... sorry, have removed it now.

 

[Noctrin]

did you manage to find the device id causing the buffer overflow, in my case, i cannot find "0xfffff801c8f631d0" listed anywhere. I assume it's virtual but i have no clue where it gets it from.

Other thing i can think of is we figure out what we're both using, find the overlap and maybe it will point something out. I'm fairly out of ideas, but as you said, it doesnt seem to cause any problems for me either, so i might just forget about it.

 

[sretalla]

No, I could never find a still-running process to match to the error.

 

[Dudleydogg]

The Error message is caused by ZoneMinder, did you guys ever resolve this? Kind of a pain to reboot the Plugin on daily basis.

 

[sretalla]

I have it an a box that doesn't run that plugin. What makes you think it's the cause?

The error doesn't seem to align with any actual problem in system performance for me.

 

[GEOptic]

freenas kernel: sonewconn are flooding here as well. 11.2U6
But I had no issues for weeks after I went from 11.1U7 to 11.2U6.

 

[anodos]

Are you using samba? I think in something like 11.1-U7 I added the ability to modify Samba's socket listen backlog by modifying the sysctl kern.ipc.soacceptqueue, and restarting samba. The default listen backlog in Samba was 50 (for smbd processes). in 11.1-U7 the change to use the sysctl value bumped it to 128. Generally, bumping up this value may help with handling sudden bursts of connections, but if the server is totally overloaded you may end up just shifting the problem somewhere else.

 

[GEOptic]

indeed it started after I started to use SMS share. But I did not make it available public...

Now, I think it's the source of my Pools not found. I changed permissions (wrongly on the pool thinking it was on the SMB share) then reset the permission thru command line chown + setfalc... then removed old unwanted share (but forgot to remove first the periodic snapshot... Et voilà... Pool disapeared.
The source, but not the fault... Now I'm trying to remove periodic snapshot thru command line (infinite turning wheel on the GUI).
Oh well... bummer... I had lots of other stuff to do...

 

[G8One2]

I get these errors if I restart the Plex jail while there are streams going at the same time. I think its because the buffer in Plex is ahead of the stream, then if you restart the jail while a stream is playing, it resets the buffer. Because Plex can continue the steam until it runs out of buffer from the server. If that's the case, stopping all streams and restarting the jail or plug in usually stops that error for me. Doesn't mean that's your issue, but might be worth a try.

 

[GEOptic]

Thanks for your input G8One2 !!! ;)

We have no jails / plug in at all!!!
I think the UI have some issues reading the configuration files … but I don't know where to double check this.
It must be SMB/AD permissions related, but then I'm not that sure ...
But since the UI won't let me do anything, I'm trying from command line! :)
Not yet found a solution … a lot of sonewconn queue overflow message ...

 

[anodos]

Thanks for your input G8One2 !!! ;)

We have no jails / plug in at all!!!
I think the UI have some issues reading the configuration files … but I don't know where to double check this.
It must be SMB/AD permissions related, but then I'm not that sure ...
But since the UI won't let me do anything, I'm trying from command line! :)
Not yet found a solution … a lot of sonewconn queue overflow message ...

Try increasing the value of the sysctl I mentioned sysctl kern.ipc.soacceptqueue=512. Once you do this, you will need to restart samba service samba_server onerestart

 

[cobrakiller58]

I have this recurring nightmare as well, one jail with a java applet running now does this once a month. I kept raising the number and stopped at 4096 I can handle restarting the jail once a month, but I would really like to figure out how to solve this....