No memory for std Rx buffers

[cpmoran]

In exploring the difference in performance to Windows on the same hardware, we came across another fault we've not been able to advance.

The machine is 'vintage' Hewlett Packard Proliant DL320s (Storage) server that has a 12 (or 14) bay LFF SAS backplane and is easily capable of sustained 100Mbytes/sec performance even though it has a max of 8Gb RAM and a crappy Xeon 3060 dual-core processor, using the Smart Array P400 controller in RAID 6. The gigabit adapters are dual Broadcom 5704 series.

Windows supports jumbo packets on the LAGGed adapter no problems for a nice performance bump.

Adding "mtu 9000" to the member adapters of the FreeNAS LAGG seems to be OK until a reboot. The server throws a message "no memory for std Rx buffers" and the links are DOWN.

We've tried Autotune and A LOT of FreeBSD networking tuning reading, specifically this page of variables and explanations.

We're kinda exhausted and wondered if there are any FreeBSD kernel/dev gurus that might have hints on where to explore what-on-earth is going on.

 

[cpmoran]

UPDATE: the issue continues with 11.3

bge0: no memory for std Rx buffers

You can set the values in the GUI, no problems and access continues.

ifconfig reports the new MTU, but on reboot the above error appears and the links are lost.

 

[cpmoran]

UPDATE: installing FreeBSD 12.1 separately on this device results in this error.

It looks like the device driver code is carried over from 2010 all the way into Release 12.1 - so I might take it to the FreeBSD forums as a bug.

Linux (Unraid) does not complain and the performance from the onboard dual Broadcom NetXtreme BCM5701s is very good. Hewlett Packard call them NC324s.

/* Init RX ring. */
if (bge_init_rx_ring_std(sc) != 0) {
device_printf(sc->bge_dev, "no memory for std Rx buffers.\n");
bge_stop(sc);
return;

 

[jgreco]

The bge driver has long been problematic in some use cases, including jumbo, especially with certain chipsets. I've had problems with it for years. Some gear works perfectly. Some doesn't.

I suggest using the Intel X520 or Chelsio cards instead.

https://www.ixsystems.com/community/resources/10-gig-networking-primer.42/

In general, jumbo frames are problematic because they are often implemented very differently than conventional frames, which fit within a 4K memory block. You are using code that isn't as robust and may be running into different kernel limits.

https://www.ixsystems.com/community/threads/jumbo-frames-notes.26064/

Your gear is old enough that I suspect jumbo would be helpful, but the problem is that there's a huge amount of effort involved in getting the driver that last ~5-10% of the way to eliminate the unusual edge cases, and unless some company is sponsoring that work, it usually doesn't happen.

FreeBSD supports lots of stuff, but when I say supports, I really mean ""supports"". If you really want reliable operation, instead of bringing random hardware and then expecting FreeBSD to work perfectly on it, what you really need to do is to use the recommended hardware guide posted here and then work forward from that.

 

[cpmoran]

If you really want reliable operation, instead of bringing random hardware and then expecting FreeBSD to work perfectly on it, what you really need to do is to use the recommended hardware guide posted here and then work forward from that.

Hewlett Packard sold tens of thousands of these servers, I wouldn't call them 'random' and I have no 'expectations'. I was simply sharing the experience in the hope that it was indeed a 'tuning' problem, or updating people where I was heading in case they came across it.

We sold at least four dozen over the years. They're capable, reliable equipment with Windows loaded, with good performance for their day. Importantly, they have 14 3.5" SAS drive bays, and parts are plentiful.

I am happy to add Intel ethernet if it is the end-of-the-road for these Broadcom controllers, but those Broadcom controllers are everywhere, probably more common ex HPE, Dell, IBM, Supermicro, Tyan etc than others except Realtek if I'd have to guess.

My goal was to re-purpose these servers so they didn't go into the IT waste stream. If I put Windows back on, they'll work like they worked before, which we can do. I was super keen to see how far they could be pushed however. Unraid seems to be the answer at the moment, which is a shame, because I really prefer the GUI work done and as ex Irix and Ultrix person, FreeBSD still interests me.