adjusting rx/tx ring buffer

[vshaulsk]

General Question as I try to tune my 10 gig network:

I have looked over the forums and general google search, but I can't seem to find a definitive answer for adjusting the rx/tx ring buffers on my Chelsio dual 10GB sfp+ NIC

From what I found it looks like in the tunables section I would add hw.cxgbe.rxd = 4096 and hw.cxgbe.txd = 4096

However before I try this, I would like to get confirmation that this is correct.


On linux I would use ethtool "ethtool -G cxgb0 rx 4096 tx 4096", but not sure how to do this in Freenas/freebsd

Also in Freenas am I able to change the ring buffer for a specific interface (cxbg0 or cxgb1) or do I have to implement the change for both interfaces at the same time


Thank you for the help

 

[vshaulsk]

With no feedback to this query it makes me think that this is not a common practice in freenas.....

Is this a common practice in freebsd ?

 

[kdragon75]

hw.cxgbe.rxd = 4096 and hw.cxgbe.txd = 4096 look like sysctl or loader variables. Searching "FreeBSD cxgbe" bring me to the manual page for the driver. from there I can look at the https://www.freebsd.org/cgi/man.cgi?query=cxgbe&sektion=4&manpath=freebsd-release-ports#end section.

 

[kdragon75]

hw.cxgbe.qsize_txq
Number of entries in a transmit queue's descriptor ring. A
buf_ring of the same size is also allocated for additional soft-
ware queuing. See ifnet(9). The default value is 1024. Differ-
ent interfaces can be assigned different values via the
dev.<port>.X.qsize_txq sysctl. This sysctl works only when the
interface has never been marked up (as done by ifconfig up).

hw.cxgbe.qsize_rxq
Number of entries in a receive queue's descriptor ring. The
default value is 1024. Different interfaces can be assigned dif-
ferent values via the dev.<port>.X.qsize_rxq sysctl. This sysctl
works only when the interface has never been marked up (as done
by ifconfig up).

So you should be able to test it live with

Code:

sysctl dev.cxgbe.0.qsize_rxq = 4096

and

Code:

sysctl dev.cxgbe.1.qsize_rxq = 4096

for cxgbe0 and cxgbe1 respectively.

 

[kdragon75]

Anytime you have a FreeBSD related question, just google FreeBSD xyz. FreeBSD has great documentation. Not to be rude but I don't like spoon feeding engineers.

 

[vshaulsk]

Thank you for the help.... I also found the manual page, but was not entirely sure I was in the right place...... hence why I asked the help of IT professionals vs a fellow vehicle suspension engineer.

 

[kdragon75]

Thank you for the help.... I also found the manual page, but was not entirely sure I was in the right place...... hence why I asked the help of IT professionals vs a fellow vehicle suspension engineer.

:p I see you updated that ;)

 

[vshaulsk]

Yes I figured I should clarify........ Mechanical automotive engineer professionally..... family business IT administrator by night :)

My other question:
- Is adjusting ring buffers common on Freenas? From my searching, I have only seen a few threads talking about it and in the 10-gig primer, I do not think I have seen it mentioned at all.

I also run proxmox hosts and on the linux side when I look at 10gig tuning adjusting ring buffers come often.
- Overall getting 10 gig networking to run at 10 gig has not gone as well as I was hoping, but I have learned a lot and it is a fun/frustrating project.

 

[kdragon75]

Yes I figured I should clarify........ Mechanical automotive engineer professionally..... family business IT administrator by night :)

My other question:
- Is adjusting ring buffers common on Freenas? From my searching, I have only seen a few threads talking about it and in the 10-gig primer, I do not think I have seen it mentioned at all.

I also run proxmox hosts and on the linux side when I look at 10gig tuning adjusting ring buffers come often.
- Overall getting 10 gig networking to run at 10 gig has not gone as well as I was hoping, but I have learned a lot and it is a fun/frustrating project.

I think the number of 10gbe users are low enough that it's not a huge topic (yet). Its worth playing with and as a sysctl, its easy enough to toggle for back to back benchmarks. I will add for virtualization, big buffers mean more throughput but typical higher latency. It all depends on if you need/want the throughput or ultra responsive virtual machines.