Elliot Dierksen
Guru
- Joined
- Dec 29, 2014
- Messages
- 1,135
I have two FreeNAS systems which are both running 11.3-U3.2. Both systems are running on Cisco C240 M4 servers with dual E5-2637 v3 CPU's. On my storage network, both systems have Chelsio T580-LP-CR nic's with 9000 MTU set. The primary server has 256G RAM, and the secondary server has 160G RAM. I am seeing some packet drops while doing testing with
I see input drops on the secondary server in the netstat output.
The switch between the two servers is a Cisco Nexus 3064, and it does not show any errors on either of the switch ports.
iperf3
. When the primary server is acting as the iperf3 server, I get 27.6 Gbits/sec throughput with a single thread, and I am ok with that. When the secondary server is acting as the iperf3 server, I get 21.5 Gbits/sec and the primary server show retries.Code:
root@freenas2:/ # iperf3 -c 192.168.252.23 Connecting to host 192.168.252.23, port 5201 [ 5] local 192.168.252.27 port 48237 connected to 192.168.252.23 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.42 GBytes 20.8 Gbits/sec 60 385 KBytes [ 5] 1.00-2.00 sec 2.50 GBytes 21.5 Gbits/sec 66 236 KBytes [ 5] 2.00-3.00 sec 2.53 GBytes 21.7 Gbits/sec 67 315 KBytes [ 5] 3.00-4.00 sec 2.51 GBytes 21.6 Gbits/sec 73 262 KBytes [ 5] 4.00-5.00 sec 2.49 GBytes 21.4 Gbits/sec 56 254 KBytes [ 5] 5.00-6.00 sec 2.64 GBytes 22.7 Gbits/sec 82 236 KBytes [ 5] 6.00-7.00 sec 2.68 GBytes 23.0 Gbits/sec 74 315 KBytes [ 5] 7.00-8.00 sec 2.54 GBytes 21.8 Gbits/sec 71 271 KBytes [ 5] 8.00-9.00 sec 2.48 GBytes 21.3 Gbits/sec 66 306 KBytes [ 5] 9.00-10.00 sec 2.49 GBytes 21.4 Gbits/sec 76 315 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 25.3 GBytes 21.7 Gbits/sec 691 sender [ 5] 0.00-10.08 sec 25.3 GBytes 21.5 Gbits/sec receiver iperf Done.
I see input drops on the secondary server in the netstat output.
Code:
root@freenas:/ # netstat -in | sed -n '1p;/cxl/p' Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll cxl0 9000 <Link#3> 00:07:43:29:fc:30 223981604 0 7389 43584296 0 0 cxl0 - 192.168.252.0 192.168.252.23 14937091 - - 42844382 - - cxl1 9000 <Link#4> 00:07:43:29:fc:38 5321792 0 0 8685039 0 0 cxl1 - 192.168.250.0 192.168.250.23 1462473 - - 8306704 - -
The switch between the two servers is a Cisco Nexus 3064, and it does not show any errors on either of the switch ports.