10GbE only after reboot? Then drops to 2Gb/s...

[fraenki]

Hi,

welcome to my networking nightmare: using iperf to test network speed, my FreeNAS 11.2-U3 is only able to saturate the 10GbE network for a very short time after reboot.

Uptime ~2 minutes:

Code:

root@freenas:~ # uptime
10:51PM  up 2 mins, 1 user, load averages: 0.57, 0.23, 0.09

root@freenas:~ # iperf -c centos-10.example.com
------------------------------------------------------------
Client connecting to centos-10.example.com, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.40 port 47394 connected with 10.0.0.24 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.8 GBytes  9.29 Gbits/sec

root@freenas:~ # iperf -c centos-02.example.com
------------------------------------------------------------
Client connecting to centos-02.example.com, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.40 port 47396 connected with 10.0.0.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.36 Gbits/sec

Not too bad, 9.3 Gb/s.

Uptime ~4 minutes:

Code:

root@freenas:~ # uptime
10:53PM  up 4 mins, 1 user, load averages: 0.65, 0.34, 0.15

root@freenas:~ # iperf -c centos-02.example.com
------------------------------------------------------------
Client connecting to centos-02.example.com, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.40 port 14864 connected with 10.0.0.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.89 GBytes  1.62 Gbits/sec

root@freenas:~ # iperf -c centos-10.example.com
------------------------------------------------------------
Client connecting to centos-10.example.com, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.40 port 43680 connected with 10.0.0.24 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.23 GBytes  7.07 Gbits/sec

Less than 2 Gb/s for host centos-02 and 7 Gb/s for host centos-10. Both far from 10GbE.
No load or traffic on the FreeNAS server.
iperf between centos-02 and centos-10 easily reaches 9.3 Gb/s.

Does this make sense to anyone?

FreeNAS hardware: Fujitsu RX200 S6, Xeon E5645, 24 GB RAM, Intel X550-T2 NIC
centos-02 hardware: Supermicro X10DRI, Xeon E5-2630L v3, 128 GB RAM, Intel X550-T2 NIC
centos-10 hardware: Fujitsu RX200 S6, Xeon E5645, 24 GB RAM, Intel X550-T2 NIC


Thanks
- Frank

 

[fraenki]

Less than 2 Gb/s for host centos-02 and 7 Gb/s for host centos-10. Both far from 10GbE.

Today I did another test, this time it reached only ~1.1 Gb/s. Wow.

I've read a lot about fake/counterfeit NICs, so I've replaced the X550-T2 NIC with a X540-T1.
This was ~2 hours ago and take a look at the new test results...

Code:

root@freenas:~ # iperf -c centos-02.example.com
------------------------------------------------------------
Client connecting to centos-02.example.com, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.40 port 26198 connected with 10.0.0.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.33 Gbits/sec

Looks like the X550-T2 was a fake/counterfeit NIC. Some quality issues may have caused the horrible performance.

I've made some photos of both the fake NIC and the replacement original Intel NIC...

As you can see the (supposedly fake) X550-T2 NIC is missing Intel branding and the yottamark, while the X540-T1 looks pretty genuine.

I have a very similar performance issue with another FreeNAS server and will replace it's NIC next week, maybe there's another fake NIC installed...


Thanks
- Frank

 

[jgreco]

Yeah, I specifically call out fake Intel cards in the 10 Gig Networking Primer ...

 

[fraenki]

Yeah, I specifically call out fake Intel cards in the 10 Gig Networking Primer ...

Thanks. Actually, I've read it before posting. Sorry, looks like I've missed the most important part.
Until today all I've heard about fake/counterfeit 10GBASE-T NICs was "it works", couldn't find a single report stating that it caused issues.


Thanks
- Frank

 

[fraenki]

Today I did another test, this time it reached only ~1.1 Gb/s. Wow.
Looks like the X550-T2 was a fake/counterfeit NIC. Some quality issues may have caused the horrible performance.

Unfortunately, the issue is NOT solved. I did severall iperf tests today and the performance issue between FreeNAS and some CentOS hosts is still there...

Code:

root@freenas:~ # uptime
11:57PM  up 3 days,  3:49, 3 users, load averages: 0.21, 0.19, 0.17

root@freenas:~ # iperf -c centos-02.example.com
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  5.61 GBytes  4.81 Gbits/sec
[  3]  0.0-10.1 sec  2.05 GBytes  1.75 Gbits/sec
[  3]  0.0-10.0 sec  5.52 GBytes  4.74 Gbits/sec
[  3]  0.0-10.0 sec  1.91 GBytes  1.64 Gbits/sec

- Frank