I've run into a weird issue over the past couple of days. I just installed a 10Gb Chelsio card (T520-SO-CR) in my FREENAS-MINI-3.0-XL+ (TrueNAS-12.0-U4). I put the same model Chelsio in a Proxmox machine as well and connected both to a Microtik 10Gb switch via some DAC cables. Since they are the only two devices on the switch, I'm using a /30 and have assigned 10.0.0.1 to TrueNAS and 10.0.0.2 to Proxmox.
When I run iperf from TrueNAS, at first I get results like this:
After a short while, less than an hour usually, every time I run iperf from TrueNAS, I get these results:
And every run continues to stay at this 1Gb rate until I do something like reboot the TrueNAS box. Then after a little bit of time, the bandwidth goes back down from 10gb to 1gb speeds showing with iperf. I don't see anything related to the NIC happening in dmesg or /var/log/messages when the slowdown starts happening, either. And the switch itself always shows the autoneg speed of 10Gb for this connection.
If I run iperf from the Proxmox machine, it always shows 10Gb speeds, so the weird slowdown just seems to be in one direction.
I checked and the cxgbe firmware version is 1.25.0.0, which I believe is the expected version for this version of TrueNAS.
I was hoping maybe someone has seen this behavior before or would know if there is anything in particular that needs to be configured for the T520 series of cards when used with TrueNAS? I'm kind of at a loss here and would appreciate any help anyone might be able to offer.
When I run iperf from TrueNAS, at first I get results like this:
Code:
iperf -c 10.0.0.2 -i1 ------------------------------------------------------------ Client connecting to 10.0.0.2, TCP port 5001 TCP window size: 1.72 MByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.1 port 41027 connected with 10.0.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.09 GBytes 9.38 Gbits/sec [ 3] 1.0- 2.0 sec 1.09 GBytes 9.40 Gbits/sec [ 3] 2.0- 3.0 sec 1.08 GBytes 9.29 Gbits/sec [ 3] 3.0- 4.0 sec 1.09 GBytes 9.36 Gbits/sec [ 3] 4.0- 5.0 sec 1.09 GBytes 9.32 Gbits/sec [ 3] 5.0- 6.0 sec 1.08 GBytes 9.28 Gbits/sec [ 3] 6.0- 7.0 sec 1.09 GBytes 9.35 Gbits/sec [ 3] 7.0- 8.0 sec 1.09 GBytes 9.34 Gbits/sec [ 3] 8.0- 9.0 sec 1.07 GBytes 9.23 Gbits/sec [ 3] 9.0-10.0 sec 1.09 GBytes 9.33 Gbits/sec [ 3] 0.0-10.0 sec 10.9 GBytes 9.33 Gbits/sec
After a short while, less than an hour usually, every time I run iperf from TrueNAS, I get these results:
Code:
iperf -c 10.0.0.2 -i1 ------------------------------------------------------------ Client connecting to 10.0.0.2, TCP port 5001 TCP window size: 96.8 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.1 port 19966 connected with 10.0.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 108 MBytes 907 Mbits/sec [ 3] 1.0- 2.0 sec 102 MBytes 858 Mbits/sec [ 3] 2.0- 3.0 sec 104 MBytes 872 Mbits/sec [ 3] 3.0- 4.0 sec 103 MBytes 861 Mbits/sec [ 3] 4.0- 5.0 sec 105 MBytes 884 Mbits/sec [ 3] 5.0- 6.0 sec 106 MBytes 885 Mbits/sec [ 3] 6.0- 7.0 sec 106 MBytes 886 Mbits/sec [ 3] 7.0- 8.0 sec 104 MBytes 875 Mbits/sec [ 3] 8.0- 9.0 sec 109 MBytes 915 Mbits/sec [ 3] 9.0-10.0 sec 105 MBytes 879 Mbits/sec [ 3] 0.0-10.0 sec 1.03 GBytes 882 Mbits/sec
And every run continues to stay at this 1Gb rate until I do something like reboot the TrueNAS box. Then after a little bit of time, the bandwidth goes back down from 10gb to 1gb speeds showing with iperf. I don't see anything related to the NIC happening in dmesg or /var/log/messages when the slowdown starts happening, either. And the switch itself always shows the autoneg speed of 10Gb for this connection.
If I run iperf from the Proxmox machine, it always shows 10Gb speeds, so the weird slowdown just seems to be in one direction.
I checked and the cxgbe firmware version is 1.25.0.0, which I believe is the expected version for this version of TrueNAS.
I was hoping maybe someone has seen this behavior before or would know if there is anything in particular that needs to be configured for the T520 series of cards when used with TrueNAS? I'm kind of at a loss here and would appreciate any help anyone might be able to offer.