Listen queue overflow: 8 already in queue awaiting acceptance

[Hostigal]

kernel log messages:
> sonewconn: pcb 0xfffff800995fa3c0: Listen queue overflow: 8 already in queue awaiting acceptance (6 occurrences)
> sonewconn: pcb 0xfffff800995fa3c0: Listen queue overflow: 8 already in queue awaiting acceptance (3 occurrences)
-- End of security output --

What does this mean, how to proceed?

 

[dlavigne]

Sounds like this: https://forums.freenas.org/index.php?threads/freenas-9-2-error-listen-queue-overflow.17774/. What type of NIC is it?

 

[keithg]

I am having a similar problem. Same hardware for 5 years or so. Same router and computers for a while. Just last week, I started getting this in my daily report over and over:
sonewconn: pcb 0xfffff8011c50a570: Listen queue overflow: 151 already in queue awaiting acceptance

It is a Dell 390 with a Broadcom 1k card:
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x00b002> mem 0
xfbdf0000-0xfbdfffff irq 17 at device 0.0 on pci2

 

[dlavigne]

I am having a similar problem. Same hardware for 5 years or so. Same router and computers for a while. Just last week, I started getting this in my daily report over and over:
sonewconn: pcb 0xfffff8011c50a570: Listen queue overflow: 151 already in queue awaiting acceptance

It is a Dell 390 with a Broadcom 1k card:
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x00b002> mem 0
xfbdf0000-0xfbdfffff irq 17 at device 0.0 on pci2

Has your use case changed? That error means that the NIC can no longer keep up.

 

[sretalla]

I have also seen these errors arrive recently (I think I would connect it with 11.1 U1, although it has continued on U2 also)

I note that it seems to happen when my Plex server is transcoding to external clients.

Unlike most others in the threads (here and the one linked in the second post), I am using an Intel NIC.

My board has 2, presented as em0 and igb0. I am using em0, which I think is the first of the two listed below:

Intel® I210-AT, 1 x Gigabit LAN Controller
Intel® I218LM, 1 x Gigabit LAN, Dual interconnect between the Integrated Media Access Controller (MAC) and Physical Layer (PHY)

Anyone else noticing it on intel NICs?

I will be doing some work to switch over to the other NIC and see if it changes anything.

 

[sretalla]

So after a little checking and some reading, it seems that the better of the two NICs on my board is the i210 (the one which uses the igb driver), so I'll be swapping that over and posting some observations after a couple of days.

I used the following (and share the output below) to find the versions for each interface:

Code:

root@freenas:~ # pciconf -lv em0
em0@pci0:0:25:0:   class=0x020000 card=0x862a1043 chip=0x15a08086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Ethernet Connection (2) I218-LM'
	class	  = network
	subclass   = ethernet
root@freenas:~ # pciconf -lv igb0
igb0@pci0:5:0:0:   class=0x020000 card=0x85571043 chip=0x15338086 rev=0x03 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'I210 Gigabit Network Connection'
	class	  = network
	subclass   = ethernet

 

[sretalla]

So in checking the other linked thread above, I see that:

To read more about kern.ipc.somaxconn mentioned in 3983 go to https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html (n.b. this sysctl is being replaced in FreeBSD 10...)

Although I see the value is changed to kern.ipc.soacceptqueue as mentioned from FreeBSD 10.

I have changed the value as recommended in the handbook to 1024 (temporarily) and will report back.

 

[sretalla]

So far after changing from em0 to igb0 and also implementing the sysctl tunable kern.ipc.soacceptqueue to 1024 (from 128), I have seen the sonewconn queue overflows continuing. (an interesting side-note, when changing kern.ipc.soacceptqueue I see the value of kern.ipc.somaxconn also changes to match)

Since it may be related to the use of my Plex server jail (runs latest plexpass version in a warden jail currently), I have considered that my update process of Plex doesn't include pkg update && pkg upgrade, so I have now done that and will report back.

 

[sretalla]

Although the issues I was having with the users of Plex from external networks have now stopped, it seems that the sonewconn queue overflow continues to be logged.

With no reason to keep chasing this other than to clean up my logs, I'm probably going to drop it as there seems to be little interest on the topic.

 

[toadman]

I've got an Intel gigabit card (em0) and previously (like at least a couple years ago) I had to change somaxconn to 2048. I've not changed it since and have had no issues. I don't see a reason you can't try a higher value. Keep doubling until the issue stops. It's a sysctl so (I think) a change in value would be reflected immediately, so no reboot needed to experiment.

 

[sretalla]

You're right, I can (and will) try that.

I read somewhere along the way that a jail needs a restart after the sysctl is changed, but should be no problem to make the change and take advantage of the next restart.

It is a little odd that the original value of 128 was doing fine for months (maybe even more than a year) and suddenly isn't enough by a factor of 10 or 20, perhaps related to still being a freebsd 10 jail (warden) running on 11.1 host. I will look at iocage when I get a chance to see if that can be of any help in the situation. (I guess I will have to put the value back to default to prove the point at the risk of upsetting my Plex guests, but such is life).

 

[sretalla]

2048 and still getting the errors.

I'm inclined to switch to iocage for Plex and see if that improves it... perhaps I will do that rather than increasing to 4096.

 

[Astrodonkey]

I'm having the same problem on my system, starting recently (after setting up iocage in conjunction with my warden jails, but not sure if this is related). The difference is yours seems to keep running and mine rolls over and dies. Potentially this could be caused by something else, but I'd like to resolve these error messages first and see if my problems go away.

I am on 11.1-stable and have been unable to update to U2 (this is another problem).

 

[sretalla]

How full are your boot sticks? (perhaps this could be the issue with upgrading?)

I don't see high numbers like yours (155 to my 8 already in the queue), so there's something to understand as to why... I think mine has always been lower numbers (with variable occurrences, also mostly low)

I am also starting to see a bunch of ARP swapping between jail interfaces with the same MAC addresses, but don't think these are related, although I do see what seems like subsequent switching without switching back... going from A to B and then from A to B again, with no indication of how we got back to A.

Code:

dmesg:
------------snip-------------
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (32 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (33 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (66 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (6 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (6 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (8 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (5 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (5 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair3b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (8 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (6 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (7 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (3 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair2b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (10 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (27 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair7b
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
arp: 192.168.X.XXX moved from 02:55:50:AA:AA:AA to 38:d5:47:BB:BB:BB on epair4b
sonewconn: pcb 0xfffff801c8f73ae0: Listen queue overflow: 16 already in queue awaiting acceptance (17 occurrences)
sonewconn: pcb 0xfffff80210d013a0: Listen queue overflow: 8 already in queue awaiting acceptance (15 occurrences)
----
EOF

Have you increased your kern.ipc.soacceptqueue value (or somaxconn)?

 

[Astrodonkey]

How full are your boot sticks? (perhaps this could be the issue with upgrading?)

About 50%. 8 GB used out of 16.

Have you increased your kern.ipc.soacceptqueue value (or somaxconn)?

I did not... but I took a look at the output from netstat and saw there were a ton of connections to the FreeNAS web-ui port from my PC. I rebooted it and cleared the connections. I will keep an eye on this, but it seems like maybe just a rogue process on my LAN.

 

[sretalla]

To add an update:

After switching to iocage jails for all of my media stuff, I saw the instances of this error decrease, but then return, so I'm still looking for a long-term fix.

I will increase the kern.ipc.soacceptqueue value a few times to see if I can get it to stop.

 

[Jeff Hallam]

To add an update:

After switching to iocage jails for all of my media stuff, I saw the instances of this error decrease, but then return, so I'm still looking for a long-term fix.

I will increase the kern.ipc.soacceptqueue value a few times to see if I can get it to stop.

Did you end up finding a resolution to your sonewconn log entries? I am seeing the same from my NAS.

 

[sretalla]

I still see them.

I'm not convinced it's any kind of real problem as I don't see anything not working from the application side.

 

[alexr]

I’ve started to see these as well. Totally different hardware.

 

[sretalla]

I still see them, but can't link them to any lack of function or visible error.