Weird 10Gb Network Problem

[-Mael-]

I just upgraded my home infrastructure to 10Gb and am seeing sub-par SMB transfer speeds between my desktop PC running Window 10 and my FreeNAS server. When I run iperf tests between the two I get ~9.4Gbits/sec but when I copy a file from an SMB share to a NVMe PCIe drive on my PC I only get about 215 MB - 350MB/s.

iperf client on on PC results

Code:

iperf.exe -c x.x.x.x -P 2 -t 30 -r -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 9864
TCP window size:  208 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to x.x.x.x, TCP port 5001 with pid 9864
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  5] local x.x.x.x port 51301 connected with x.x.x.x port 5001
[  4] local x.x.x.x port 51300 connected with x.x.x.x port 5001
[ ID] Interval        Transfer    Bandwidth       Write/Err
[  5] 0.00-30.00 sec  16.6 GBytes  4.75 Gbits/sec  1/0
[  4] 0.00-30.00 sec  16.3 GBytes  4.67 Gbits/sec  1/0
[SUM] 0.00-30.00 sec  32.9 GBytes  9.42 Gbits/sec  2/0

iperf on server results

Code:

iperf -s -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 35021
Read buffer size:  128 KByte
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local x.x.x.x port 5001 connected with x.x.x.x port 51300
[  5] local x.x.x.x port 5001 connected with x.x.x.x port 51301
[ ID] Interval        Transfer    Bandwidth       Reads   Dist(bin=16.0K)
[  4] 0.00-30.01 sec  16.3 GBytes  4.67 Gbits/sec  1022122    417223:536483:62233:2831:2926:199:192:35
[  5] 0.00-30.01 sec  16.6 GBytes  4.75 Gbits/sec  1009944    395970:529811:78238:2604:2948:202:134:37
[SUM] 0.00-30.01 sec  32.9 GBytes  9.42 Gbits/sec  2032066    813193:1066294:140471:5435:5874:401:326:72

But when I copy large files from a SMB share to my PC I see the following: For this transfer let it run until it was about 30% complete

When I copy the same file a second time

You'll notice that the first 30% of the transfer runs at 10Gb/s then drops back to 218 MB/s second for the rest of the file. If I let the copy finish and copy the same file a third time, the entire file transfers 10Gb/s. Based on the results of the iperf tests, the network is capable 10Gbs, but I'm baffled by the SMB performance. Why would copying the file a second time be faster? Is the file getting cached somewhere on the FreeNAS side so it doesn't have to be pulled from disk? Does anyone have any suggestions for getting better SMB transfer speeds?


INFRASTRUCTURE DETAILS

FreeNAS Server specs are in my signature. The 10Gb card in the server is a iXsystems FreeNAS Mini XL+ 10Gb Dual-Port Upgrade SFP+ card. These are private-labeled Chelsio 520-SO-CR cards. The card is plugged into a PCIe 3.0 8x slot on the motherboard. The 12 drives are divided evenly between 2 RAIDZ2 vdevs

PC Specs: AMD 3950X CPU, 32GB RAM 2x M.2 Gen 4 NVME SSD Drives. The 10Gb network card is an Intel X550-T2

Both the FreeNAS server and the PC are connected to a Ubiquiti Networks US-16-XG. Jumbo frames are enabled on all the switches and the packet size is set to 9014 on both the PC and server.

I've tried several tuning options on the PC and server side and nothing seems to have any effect on the SMB performance.

My SMB Aux Parameters are

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no
strict sync = no

I have not added any system tunables.

 

[Samuel Tai]

I only get about 215 MB - 350MB/s.... The 12 drives are divided evenly between 2 RAIDZ2 vdevs

What you're seeing is the max your drives can push. Remember, with RAIDZ2, each sector is only stored on a single drive, with 2 other drives storing the parity of that sector. With 2 vdevs, essentially you're only getting 2 drives at any one time providing data for the copy.

When you stop each copy and then restart, you're pushing that data into ARC and L2ARC. Subsequent copies of the same data are coming out of cache onto the network, which explains the higher speed.

Now that you're on 10G, you need to consider replacing your spinning disks with SSDs to take advantage of the faster speed.

 

[Elliot Dierksen]

If iperf gives you proper results (90%+ link utilization), then it likely isn't a network problem.You should do some IO tests on your pool locally and see what kind of results you get. I suspect you won't get much more than that locally. If IOPS is the most important thing to you, you could redo your pool as 6 mirror devs. SMB doesn't normally use sync writes, so I don't think an SLOG would help you. You are only as fast as the slowest component, and you have moved around where that is with the 10G NIC cards.

 

[-Mael-]

Thanks for the quick replies... I suspected the limitation might be with the FreeNAS disk pool. I'll run some IO tests and see what I get, but I think both of you are right... I'm probably getting the maximum performance out of my existing disk subsystem.

@Samuel Tai - Thanks for explaining why I was getting better results on the subsequent transfers of the same file.

 

[-Mael-]

I've run performance tests using iozone and dd. I did quite a bit of reading about testing with both utilities, but I'm still not 100% certain I'm running the tests with the right options. The testing results seem to indicate that my disk subsystem is capable of a lot more performance than what I'm seeing in my SMB copy tests. Please let me know if I'm not testing with the right parameters or have misinterpreted the results.

IOZONE tests

For iozone I ran a few variations of the following command adjusting for file size and record size

Code:

iozone -R -l 4 -u 4 -r 1024k -s 10G -F /mnt/StorageDump/Dropbox/test1 /mnt/StorageDump/Dropbox/test2 /mnt/StorageDump/Dropbox/test3 /mnt/StorageDump/Dropbox/test4 | tee -a /tmp/iozone_results2.txt

Here are the results

	10G file 1024K record size	10G file 1024K record size	32G file 1024K record size	32G file 128K record size
	MB/s	MB/s	MB/s	MB/s
Initial write	5033	5505	4752	5120
Rewrite	5097	4165	4804	4798
Read	15385	15302	14175	16480
Re-read	15917	15253	14434	17413
Reverse Read	17224	16064	15850	16573
Stride read	16991	16677	14541	16014
Random read	16807	15677	14645	15990
Mixed workload	11793	11837	10830	11528
Random write	3288	4557	4217	3446
Pwrite	5293	4864	4853	4800
Pread	15396	14141	14921	15604
Fwrite	5120	4930	4830	4391
Fread	11475	11569	11329	13542

DD TESTS

with /dev/random

Code:

dd if=/dev/random of=test1 bs=1M count=32000
dd if=test1 of=/dev/null bs=1M count=32000

Write = 182 MB/s
Read = 4504 MB/s

with /dev/zero

Code:

dd if=/dev/zero of=test2 bs=1M count=32000
dd if=test2 of=/dev/null bs=1M count=32000

Write = 5552 MB/s
Read = 8707 MB/s

 

[Samuel Tai]

Try rerunning iozone with -e (flush) -I (bypass buffers). As it is, too much of your test is using buffers and cache.

 

[-Mael-]

I tried re-running the iozone tests with the -e and -I options and the test never finishes. I even lowered the file size down to 1G and it never finishes. In the top output I could see 4 iozone processes running each using ~100% of the CPU for the process (still only 37% overall CPU,utilization). It seems to be hanging on the last freaders step.

 

[SweetAndLow]

You need to test without compression turned on. Also testing with /dev/random really only tests your cpu not your disks, lol. You need to use /dev/zero

 

[-Mael-]

After fighting with iozone for a bit I finally got the tests to run with the -I and -e options. I ended up dropping the fread & Re-fread test since it was hanging when I ran the test with -I. To @SweetAndLow's suggestion I ran these against a file system with compression turned off.

Code:

iozone -R -i 0 -i 1 -i 2 -i 3 -i 4 -i 5 -i 6 -i 8 -i 9 -i 10 -i 11 -i 12 -I -e -l 2 -u 2 -r 1024k -s 10G -F /mnt/StorageDump/Footage/test1 /mnt/StorageDump/Footage/test2 /mnt/StorageDump/Footage/test3 /mnt/StorageDump/Footage/test4 | tee -a /mnt/StorageDump/Dropbox/iozone_results10.txt

	10G file 1024K record size (no cache and buffer options -e -I) (skipped fread) No Compression
Initial write	472
Rewrite	476
Read	7471
Re-read	5462
Reverse Read	7091
Stride read	8467
Random read	8435
Mixed workload	4437
Random write	425
Pwrite	494
Pread	8195
Fwrite	369
Fread

DD tests against the uncompressed file system

Code:

dd if=/dev/zero of=test2 bs=1M count=32000
dd if=test2 of=/dev/null bs=1M count=32000

Write = 616 MB/s
Read = 4417 MB/s

Writes are on par with what I've seen on my SMB testing, but reads (aka copying a file to my PC from the NAS) is still MUCH slower than any iozone read test. It still looks like my disk subsystem is capable of higher performance for read transactions than what I'm getting in the real world. I wonder if there are any additional SMB tuning changes or other tweaks I can make to speed things up?

 

[DrewN]

You should be able to get a little more performance. I’m by no means a tuning expert, but have done a fair amount of testing and tuning.

I’ve noticed lower performance with my T5 Chelsio NIC’s as opposed to my T4 Chelsio’s and Mellanox NIC’s. Why? I have no idea, but it’s a fact.

you may want to experiment with turning off/on flow control and tcp offloading. There have been scenarios where turning one or both off on server and on client has increased performance, there have been occasions when no difference too.

Sounds like you did all the kernel tuning. I experimented with kernel tuning on server and client, further than the basic recommended tuning parameters, and had some success. it’s can sometimes be hard to isolate the parameter changes that have the biggest impact, when there are so many variables and so many parameters. Your testing process is key to figuring out what does and doesn’t work.

Also consider turning off SMB signing, if it poses no security risk to you, then try it out.

I generally got tired of tuning and tweaking SMB for both MacOS and Windows, so I now pretty much exclusively use FTP when I need max file transfer throughput. I’m the only user of my system, so it’s not a security issue for me. When I use FTP, my 10GbE and 40GbE lines run fully saturated, IOPS are max, and it makes me happy, haha. For me, since I can run wide open over FTP, it tells me that the issues i have, that many people have with SMB, NFS, AFP, must be due to the file sharing parameters, likely security parameters, and not much to do with kernel tuning, disk subsystem performance, or NIC tuning. I guess that’s also verified when testing with utilities such as iperf. I’m not knowledgeable enough in file sharing and networking to know why. I just know on my all-flash systems, I get decent windows SMB speed, subpar AFP and SMB speed on MacOS, mediocre NFS speed on Windows, MacOS, and Linux, and above average iSCSI performance with all OS’. But, the security settings or lack there of with FTP presents a wide open lane for data transfer. I can fully saturate 40GbE line with large sequential file transfers. For transferring high volume small files, I can push my NvME array near its max capabilities.