SOLVED Can't get full 20Gb/s with Intel X520 da2

[Jeff Chen]

I'm still new to FreeNAS but couldn't resist to pimp my box with a 10Gb network adapter. So I got a pair of Intel X520 da2, put one in my FreeNAS box, and the other in my windows machine.

I connected both sides with two fiber optics, and set up different subnets on each pair of the connection for separation. Then I tried to run two instances of iperf3 server/client on these machines to see if they could handle 20Gb/s total throughput.

The result was a constant 13Gb/s in total single direction, as shown in the screenshot:

As soon as the 2nd instance on the other pair of ports started, the speed of both pairs equalized to 7Gb/s. And when the first instance finished, the 2nd instance jumped to full 10Gb/s.

I've tried all kinds of optimizations on both ends (tunables, driver adjustments) but this number doesn't change. I've tested the other way and it was about the same numbers.

Being curious, I tested again with two instances in different directions simultaneously, and the card handled it well:

==========================
Aaaaaand just as I was composing the post and gathering hardware information, I realized the card on my Windows machine was only working at PCIe 2.0 4x, which only has like 16Gb/s throughput. Guess I found the bottleneck on my own after all...

 

[kdragon75]

This is a great post! This demonstrates the need to look closely at underlying hardware and the need to include things like the CPU and motherboard in posts about performance.

Thank you for the update!

 

[Jeff Chen]

This is a great post! This demonstrates the need to look closely at underlying hardware and the need to include things like the CPU and motherboard in posts about performance.

Thank you for the update!

I'm thrilled that you didn't find my post laughable :)

I agree that when troubleshooting a problem, you always have to get the full picture and dig deep at the same time. Every bit of information counts.

As a response I have added my environment to the signature like everybody does. Still too new to this forum!