Tuning my FreeNAS -- iperf shows immediate drop

[trane]

Hello,

I hope this is the correct subforum for my question, but here goes. I've started tuning my brand new FreeNAS install as I've already noticed some occasional blips in copying files to my device. Anyways, to begin tuning, I started with the network connection. The program iperf shows some curious behaviour with my FreeNAS box, where it starts off around 1Gbit/sec performance and steeply drops off to an average of 5mbit/sec. I have no idea what's going on and couldn't immediately find similar stories by searching. Hopefully someone here can point me in the right direction to fix this.

First, some details:

FreeNAS-9.1.0-RC1-x64 (e3440c6)
Intel Q9300 CPU
8GB DDR2 RAM
2xIntel Pro/1000 NIC
3x3TB Seagate HDD
DLink DGS-1210 switch

When I first ran iperf between my desktop machine and the FreeNAS box, I noticed very strange behaviour, with 1 second intervals. The connection starts off at 1Gbit but drops off after about 10-15 seconds:

Code:

C:\Users\trane\Downloads>iperf -c 172.23.23.108 -P 1 -i 1  -t 50
------------------------------------------------------------
Client connecting to 172.23.23.108, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[156] local 172.23.23.60 port 54080 connected with 172.23.23.108 port 5001
[ ID] Interval      Transfer    Bandwidth
[156]  0.0- 1.0 sec    105 MBytes    877 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  1.0- 2.0 sec    106 MBytes    892 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  2.0- 3.0 sec    107 MBytes    895 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  3.0- 4.0 sec    112 MBytes    939 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  4.0- 5.0 sec    102 MBytes    857 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  5.0- 6.0 sec    105 MBytes    884 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  7.0- 8.0 sec    105 MBytes    880 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  8.0- 9.0 sec    112 MBytes    936 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  9.0-10.0 sec    102 MBytes    857 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 10.0-11.0 sec    648 KBytes  5.31 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 11.0-12.0 sec    512 KBytes  4.19 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 12.0-13.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 13.0-14.0 sec  1.55 MBytes  13.0 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 14.0-15.0 sec    512 KBytes  4.19 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 15.0-16.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 16.0-17.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 17.0-18.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 18.0-19.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 19.0-20.0 sec  1.55 MBytes  13.0 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 20.0-21.0 sec  3.27 MBytes  27.4 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 21.0-22.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 22.0-23.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 23.0-24.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 24.0-25.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 25.0-26.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 26.0-27.0 sec  4.02 MBytes  33.8 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 27.0-28.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 28.0-29.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 29.0-30.0 sec  4.08 MBytes  34.2 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 30.0-31.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 31.0-32.0 sec  1.67 MBytes  14.0 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 32.0-33.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 33.0-34.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 34.0-35.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 35.0-36.0 sec  3.46 MBytes  29.0 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 36.0-37.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 37.0-38.0 sec  1.54 MBytes  12.9 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 38.0-39.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 39.0-40.0 sec  4.23 MBytes  35.5 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 40.0-41.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 41.0-42.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 42.0-43.0 sec  1.49 MBytes  12.5 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 43.0-44.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 44.0-45.0 sec    768 KBytes  6.29 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 45.0-46.0 sec  1.46 MBytes  12.3 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 46.0-47.0 sec    704 KBytes  5.77 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 47.0-48.0 sec    576 KBytes  4.72 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 48.0-49.0 sec  3.32 MBytes  27.9 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156] 49.0-50.0 sec    640 KBytes  5.24 Mbits/sec
[ ID] Interval      Transfer    Bandwidth
[156]  0.0-50.0 sec  1.09 GBytes    187 Mbits/sec

Repeating the test with 10s intervals and for longer periods showed the same results.


As a benchmark, I ran iperf between my desktop machine and a VM on my ESXi server and saw sustained 1GB/sec preformance:

Code:

C:\>iperf -c 192.168.0.120 -P 1 -i 1   -t 50
------------------------------------------------------------
Client connecting to 192.168.0.120, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[156] local 192.168.0.60 port 54902 connected with 192.168.0.120 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0- 1.0 sec    111 MBytes    933 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  1.0- 2.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  2.0- 3.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  3.0- 4.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  4.0- 5.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  5.0- 6.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  6.0- 7.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  7.0- 8.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  8.0- 9.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  9.0-10.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 10.0-11.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 11.0-12.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 12.0-13.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 13.0-14.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 14.0-15.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 15.0-16.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 16.0-17.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 17.0-18.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 18.0-19.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 19.0-20.0 sec    113 MBytes    947 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 20.0-21.0 sec    113 MBytes    950 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 21.0-22.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 22.0-23.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 23.0-24.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 24.0-25.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 25.0-26.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 26.0-27.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 27.0-28.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 28.0-29.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 29.0-30.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 30.0-31.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 31.0-32.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 32.0-33.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 33.0-34.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 34.0-35.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 35.0-36.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 36.0-37.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 37.0-38.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 38.0-39.0 sec    113 MBytes    947 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 39.0-40.0 sec    113 MBytes    950 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 40.0-41.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 41.0-42.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 42.0-43.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 43.0-44.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 44.0-45.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 45.0-46.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 46.0-47.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 47.0-48.0 sec    113 MBytes    949 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 48.0-49.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156] 49.0-50.0 sec    113 MBytes    948 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
 
[156]  0.0-50.0 sec  5.52 GBytes    948 Mbits/sec

I have tried swapping in a different patch cable, but it did not make a difference.

Can anyone shed some light onto why I'm seeing such poor network performance?

 

[cyberjock]

First thing I'd do is upgrade to 9.1.0-RELEASE...

Also please include any sysctls and tunables you have on your system that are custom.

 

[trane]

Had no customs or tunables (just starting to tune). However, upgrading to 9.1.0-RELEASE seems to have resolved this issue.

 

[cyberjock]

Well, that was easy!

I will warn you that tuning is far from trivial. Even experts can spend weeks trying various settings to get exactly what works. Generally speaking, you shouldn't need to do any tuning except if you have non-standard uses or use iSCSi on ZFS or use an ESXi store on FreeNAS via NFS.