FreeNAS 11.1-U6 10GigE writes, 150MB/sec reads (iperf is fine)

[acquacow]

The parameters are in that image I posted above.. give it a shot, it can't hurt. :)

I've done 90% of those a few times on the last install with no change, maybe one or two of them will be magic. I assume I gave to change the igb ones to ix since that is my 10Gig chipset?

Sent from my Moto Z (2) using Tapatalk

 

[c32767a]

I've done 90% of those a few times on the last install with no change, maybe one or two of them will be magic. I assume I gave to change the igb ones to ix since that is my 10Gig chipset?

Derp, that leaked in from an old configuration. igb is an intel 1G card. You can disregard those 2 lines.

 

[acquacow]

So here's the main question, should I be able to max out 10gige with a single iperf thread? I seem to get mixed answers about that.

A single thread gets me 1.28Gbit, but I can run 8 of them and get close to line speed.

I haven't seen iperf or iperf3 on any of my hardware push a full 10gig on a single thread I don't think...

Actually I take that back, I just popped up an iperf3 server in a centos VM and my freenas box hits that at 9.15Gbit on a single thread.

So maybe something is fubar with my windows 10gig nic driver.

 

[downingjosh]

So here's the main question, should I be able to max out 10gige with a single iperf thread? I seem to get mixed answers about that.

A single thread gets me 1.28Gbit, but I can run 8 of them and get close to line speed.

I haven't seen iperf or iperf3 on any of my hardware push a full 10gig on a single thread I don't think...

Actually I take that back, I just popped up an iperf3 server in a centos VM and my freenas box hits that at 9.15Gbit on a single thread.

So maybe something is fubar with my windows 10gig nic driver.

You're running into the performance limitations of iperf on Windows, it's just not as performant as it has to go through cygwin libraries. I keep a Ubuntu VM on-hand for iperf testing for exactly this reason (you can spin them up on Amazon EC2 to test your WAN throughput easily too, btw).

 

[acquacow]

I still don't understand why SMB and FTP speeds increase 2-3x when I have the 2nd 10gige interface enabled, esp when it's not even on the same subnet.

 

[Johnnie Black]

So here's the main question, should I be able to max out 10gige with a single iperf thread? I seem to get mixed answers about that.

I believe so and that's part of your problem, I can get close to max on a single thread, I also get close to line speed transferring to/from FreeNAS, I'm using Mellanox ConnectX-2 NICs on both Windows and FreeNAS.

Iperf2, Windows as client:

Code:

Client connecting to 10.0.0.5, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.90 port 52807 connected with 10.0.0.5 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.1 GBytes  9.57 Gbits/sec

FreeNAS as client:

Code:

Client connecting to 10.0.0.90, TCP port 5001
TCP window size: 2.01 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.5 port 24075 connected with 10.0.0.90 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  10.1 GBytes  8.67 Gbits/sec

 

[c32767a]

I still don't understand why SMB and FTP speeds increase 2-3x when I have the 2nd 10gige interface enabled, esp when it's not even on the same subnet.

It's a mystery to me. The only explanation I can think of is a software problem, perhaps an interrupt limit or resource allocation that's shared, but doubled in size when the 2nd interface is enabled, therefore providing additional resources to the single active port?

I can only say we run intel cards on 11.1U5 and for NFS and AFP, we see close to wire speed on 10G doing IO to an all flash pool with only 1 ix based nic in operation.
I don't have any Windows VMs or SMB to test on our flash arrays, but this just seems weird.

 

[acquacow]

Swapped to a nice, shielded run of cat6e and no change in speeds, so that rules out my wiring.

 

[acquacow]

Welp, I dug up two articles, and one of the two, or both fixed my speeds.

https://blogs.technet.microsoft.com/netgeeks/2018/05/21/a-word-about-autotuninglevel-tcp-receive-auto-tuning-level-explained/

https://www.omelsoft.com/fix-slow-internet-windows-10-creators-update/

I still have FreeNAS 100% default tuning-parameters-wise, but I'm cranking a full 1GB/sec for SMB file transfers in both directions now without issue.

My single thread of iperf3 still isn't a full 9.5Gbit/sec, but it's way better than the 1.28Gbit/sec I was getting:


[root@freenas ~]# iperf3 -c 192.168.2.53
Connecting to host 192.168.2.53, port 5201
[ 5] local 192.168.2.237 port 19688 connected to 192.168.2.53 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 412 MBytes 3.46 Gbits/sec 0 210 KBytes
[ 5] 1.00-2.00 sec 411 MBytes 3.44 Gbits/sec 0 210 KBytes
[ 5] 2.00-3.00 sec 398 MBytes 3.34 Gbits/sec 0 210 KBytes
[ 5] 3.00-4.00 sec 411 MBytes 3.45 Gbits/sec 0 210 KBytes
[ 5] 4.00-5.00 sec 403 MBytes 3.38 Gbits/sec 0 210 KBytes
[ 5] 5.00-6.00 sec 406 MBytes 3.41 Gbits/sec 0 210 KBytes
[ 5] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec 0 210 KBytes
[ 5] 7.00-8.00 sec 410 MBytes 3.44 Gbits/sec 0 210 KBytes
[ 5] 8.00-9.00 sec 405 MBytes 3.39 Gbits/sec 0 210 KBytes
[ 5] 9.00-10.00 sec 410 MBytes 3.44 Gbits/sec 0 210 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec receiver

iperf Done.

[root@freenas ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.53, port 50044
[ 5] local 192.168.2.237 port 5201 connected to 192.168.2.53 port 50045
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 547 MBytes 4.59 Gbits/sec
[ 5] 1.00-2.00 sec 672 MBytes 5.64 Gbits/sec
[ 5] 2.00-3.00 sec 658 MBytes 5.52 Gbits/sec
[ 5] 3.00-4.00 sec 670 MBytes 5.62 Gbits/sec
[ 5] 4.00-5.00 sec 677 MBytes 5.68 Gbits/sec
[ 5] 5.00-6.00 sec 675 MBytes 5.67 Gbits/sec
[ 5] 6.00-7.00 sec 675 MBytes 5.66 Gbits/sec
[ 5] 7.00-8.00 sec 670 MBytes 5.62 Gbits/sec
[ 5] 8.00-9.00 sec 677 MBytes 5.68 Gbits/sec
[ 5] 9.00-10.00 sec 674 MBytes 5.65 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.44 GBytes 5.53 Gbits/sec receiver


 

[purduephotog]

Welp, I dug up two articles, and one of the two, or both fixed my speeds.

https://blogs.technet.microsoft.com/netgeeks/2018/05/21/a-word-about-autotuninglevel-tcp-receive-auto-tuning-level-explained/

https://www.omelsoft.com/fix-slow-internet-windows-10-creators-update/

I still have FreeNAS 100% default tuning-parameters-wise, but I'm cranking a full 1GB/sec for SMB file transfers in both directions now without issue.

My single thread of iperf3 still isn't a full 9.5Gbit/sec, but it's way better than the 1.28Gbit/sec I was getting:


[root@freenas ~]# iperf3 -c 192.168.2.53
Connecting to host 192.168.2.53, port 5201
[ 5] local 192.168.2.237 port 19688 connected to 192.168.2.53 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 412 MBytes 3.46 Gbits/sec 0 210 KBytes
[ 5] 1.00-2.00 sec 411 MBytes 3.44 Gbits/sec 0 210 KBytes
[ 5] 2.00-3.00 sec 398 MBytes 3.34 Gbits/sec 0 210 KBytes
[ 5] 3.00-4.00 sec 411 MBytes 3.45 Gbits/sec 0 210 KBytes
[ 5] 4.00-5.00 sec 403 MBytes 3.38 Gbits/sec 0 210 KBytes
[ 5] 5.00-6.00 sec 406 MBytes 3.41 Gbits/sec 0 210 KBytes
[ 5] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec 0 210 KBytes
[ 5] 7.00-8.00 sec 410 MBytes 3.44 Gbits/sec 0 210 KBytes
[ 5] 8.00-9.00 sec 405 MBytes 3.39 Gbits/sec 0 210 KBytes
[ 5] 9.00-10.00 sec 410 MBytes 3.44 Gbits/sec 0 210 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec receiver

iperf Done.

[root@freenas ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.53, port 50044
[ 5] local 192.168.2.237 port 5201 connected to 192.168.2.53 port 50045
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 547 MBytes 4.59 Gbits/sec
[ 5] 1.00-2.00 sec 672 MBytes 5.64 Gbits/sec
[ 5] 2.00-3.00 sec 658 MBytes 5.52 Gbits/sec
[ 5] 3.00-4.00 sec 670 MBytes 5.62 Gbits/sec
[ 5] 4.00-5.00 sec 677 MBytes 5.68 Gbits/sec
[ 5] 5.00-6.00 sec 675 MBytes 5.67 Gbits/sec
[ 5] 6.00-7.00 sec 675 MBytes 5.66 Gbits/sec
[ 5] 7.00-8.00 sec 670 MBytes 5.62 Gbits/sec
[ 5] 8.00-9.00 sec 677 MBytes 5.68 Gbits/sec
[ 5] 9.00-10.00 sec 674 MBytes 5.65 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.44 GBytes 5.53 Gbits/sec receiver


Thank you for the articles to read.

Just to be sure I understand- you're getting 1GB/s up/down to the SSD, and ~500MB/s for the spinnies? Any other tidbits you can offer?

Thanks much!

 

[acquacow]

Pretty much everything is in this thread.

I maxed my nic buffers, used good networking gear, disabled atime on my filesystems, have multi-channel enabled in SMB....