SOLVED Slow transfer speed when reading from FreeNAS to local PC

[Zofoor]

I have some strange problem when reading from a samba share to my local PC, and I really don't have any clue on what could be the issue.

The problem is that I get really slow data transfer from FreeNAS samba share to my PC, slower than 50MB/sec.

I have done some tests to find the issue:
Network test, from FreeNAS to my PC and from my PC to FreeNAS

root@freenas:~ # iperf3 -c 192.168.91.13 -t 30 -i 5 -p 5201
Connecting to host 192.168.91.13, port 5201
[ 5] local 192.168.91.201 port 23091 connected to 192.168.91.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-5.00 sec 566 MBytes 950 Mbits/sec 0 208 KBytes
[ 5] 5.00-10.00 sec 566 MBytes 949 Mbits/sec 0 208 KBytes
[ 5] 10.00-15.00 sec 566 MBytes 949 Mbits/sec 0 208 KBytes

root@freenas:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.91.13, port 57764
[ 5] local 192.168.91.201 port 5201 connected to 192.168.91.13 port 57765
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 107 MBytes 895 Mbits/sec
[ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 5] 2.00-3.00 sec 113 MBytes 949 Mbits/sec
[ 5] 3.00-4.00 sec 113 MBytes 949 Mbits/sec
[ 5] 4.00-5.00 sec 113 MBytes 949 Mbits/sec
[ 5] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec
[ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec

So, does not seems a network problem.

Then, from a dataset with compression disabled:

root@freenas:/mnt/main_volume/performanceTest # dd if=/dev/zero of=test.dat bs= 2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 48.916692 secs (428719095 bytes/sec)
root@freenas:/mnt/main_volume/performanceTest # dd of=/dev/null if=test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 3.323720 secs (6309652434 bytes/sec)

So, seems pretty fast on reading and writing from/to the dataset...

I try to copy a file from the same PC to FreeNAS I get a stable 113MB/s

And now the problem, transfer of a file from Win10 PC to FreeNAS using Samba:
starts at about 110MB/sec and then drops to 44-50 Mb/sec.

I have no idea on what it could be. It's not the HD of the PC is an Samsung Evo 860, I can clone a file from a folder to another at much faster speed.

Then I made another test, from Freenas using an nginx webserver to my local PC using Chrome. I get about 33-40MB/sec, so again very slow performance.

These are my tunables:

Build FreeNAS-11.1-U6
Platform Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz
Memory 65316MB

The PC was the only device connected when I was making the test. 1Gb ethernet. CPU at less than 10%, dataset at 59%, 6 disks raidz2 with 6x6Tb WD Red.

Anyway got any idea? It's really frustrating.

edit:

root@freenas:/mnt/main_volume/media # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 14.9G 5.79G 9.09G - - 38% 1.00x ONLINE -
main_volume 32.6T 19.4T 13.2T - 18% 59% 1.00x ONLINE /mnt

root@freenas:/mnt/main_volume/media # zpool status main_volume
pool: main_volume
state: ONLINE
scan: resilvered 3.24T in 0 days 08:56:45 with 0 errors on Thu Oct 4 03:23:06 2018
config:

NAME STATE READ WRITE CKSUM
main_volume ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/ca758fcd-6eb8-11e6-aa9a-0cc47acb787c.eli ONLINE 0 0 0
gptid/cb252f4b-6eb8-11e6-aa9a-0cc47acb787c.eli ONLINE 0 0 0
gptid/cbd50ee0-6eb8-11e6-aa9a-0cc47acb787c.eli ONLINE 0 0 0
gptid/cc8f75c2-6eb8-11e6-aa9a-0cc47acb787c.eli ONLINE 0 0 0
gptid/cd49df88-6eb8-11e6-aa9a-0cc47acb787c.eli ONLINE 0 0 0
gptid/f19dda6f-c728-11e8-ae03-0cc47acb787c.eli ONLINE 0 0 0

errors: No known data errors

(resilvered yesterday becouse of an HD replacement).

 

[HoneyBadger]

What performance do you get if you dd if=/mnt/main_volume/path/to/testFile.rar of=/dev/null bs=1M count=4K?

This should read the first 4GB of that file and pipe it to /dev/null - if it's still slow, the issue is with your disk subsystem or possibly a badly fragmented file.

 

[Zofoor]


root@freenas:/mnt/main_volume/media # dd if=/mnt/main_volume/media/testFile.rar of=/dev/null bs=1M count=4K
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.596475 secs (447556754 bytes/sec)


NOTE: On second execution, as the file get cached, I got
4294967296 bytes transferred in 0.708842 secs (6059129277 bytes/sec)

 

[HoneyBadger]

Hmm, so not a disk issue, your iperf seems to imply the network is okay, it's not strictly SMB because running an nginx server on FreeNAS and then pushing it over HTTP still gives you the same poor speeds ...

Very odd.

Do you have another system that might be able to check?

 

[Zofoor]

I feel so stupid... sometimes seems that FreeNAS and Linux are easier than Windows... at least when you have an error for sure you have tons of log about it.

Yes, you have found the point, with another much older system I got much better bandwidth. I don't know why I get so bad results on my workstation, it was going perfect a few time ago.. seems I have to investigate somewhere else!

 

[HoneyBadger]

I don't know why I get so bad results on my workstation

I want to say it's related to Windows 10 - I recall reading something in another thread about the TCP receive window size being improperly sized by default. (But I don't use Windows 10 at home, so I can't verify this right now.)

https://blogs.technet.microsoft.com...evel-tcp-receive-auto-tuning-level-explained/

 

[runevn]

@Zofoor, did you find a solution on this issue? I'm experiencing the same thing with windows 10 - fast upload speed (100MB/s) but slow download speed (3-11MB/s).

 

[Dariusz1989]

+1 to the issue for me too. I did format, before format I was 500mb/up/down now I'm 500+ up and 40 download. I remember spending a lot of time in windows to change some registrys to make it work before but now I'm lost sigh... damn windows.

Readin @HoneyBadger link, it looks like thats something I had to change before to make it go fast sigh... gotta go re-read all articles again I see...

 

[Constantin]

I hope it isn't an SMR drive on your local PC that is slowing things down! That could help explain the fast upload but slow download speeds... especially when you trigger a CMR cache to SMR sector transfer. So I'd expect unsteady performance - fast at first, then much slower.

I hear there are similar issues with some consumer SSDs that have a fast cache up front and much slower flash in the back - once the cache fills, the performance drops a lot. So I wonder if you could duplicate a large file locally to confirm that the local disk can write a big file fast w/o issues.

 

[Dariusz1989]

I hope it isn't an SMR drive on your local PC that is slowing things down! That could help explain the fast upload but slow download speeds... especially when you trigger a CMR cache to SMR sector transfer. So I'd expect unsteady performance - fast at first, then much slower.

I hear there are similar issues with some consumer SSDs that have a fast cache up front and much slower flash in the back - once the cache fills, the performance drops a lot. So I wonder if you could duplicate a large file locally to confirm that the local disk can write a big file fast w/o issues.

Hmm interesting points but I'm afraid its not that... Since before format all was "fast" and now its slow. Its down to windos config, my sfp+ style connection and some network settings. My workstation disks are m.2 and they do push good speeds as they should.

I think its windows, because 3 days ago it was going fast on windows :- ) I didn't change nas stuff I changed win stuff.

EDIT. Actually it made me wonder if its nas issue... as windows to Nas is fast, and I take windows properly name their packets/tag and send & nas reads it properly. But maybe nas does something wrong when it send packets back to windows and then the win10 just fails? Could it be smb from nas?

 

[mjflower]

I found this I had exactly the same problem. I had done a test install of Freenas using the inbuilt 1Gb card was getting 100-130MB transfers and verifys around the same speed from a Win 10 machine. My intel 10GBE-T card arrived today and I put it in and the transfer speeds (write) went up asI had hoped and expected but the verify on (read) went down to 5MBs! Not what I was hoping!

I found this thread and it may be totally unrelated but I remembered I had had some funny networking issues before when I first started using 10GbE on my network with a MicroTik 10GbE Switch. I logged into the switch and saw that it had some how reverted to RouterOS again and not SwitchOS. Anyway, I switched it back to boot in the SwitchOS and then my verify speeds were higher that the writes as expected. I also switched the user account to "microsoft account' as the same time so don't know if it may actually be related to that. Unlikely but thought I'd throw it in there as things to try.

As I said, it could be totally unrelated but you never know :)

Cheers

 

[anodos]

I found this I had exactly the same problem. I had done a test install of Freenas using the inbuilt 1Gb card was getting 100-130MB transfers and verifys around the same speed from a Win 10 machine. My intel 10GBE-T card arrived today and I put it in and the transfer speeds (write) went up asI had hoped and expected but the verify on (read) went down to 5MBs! Not what I was hoping!

I found this thread and it may be totally unrelated but I remembered I had had some funny networking issues before when I first started using 10GbE on my network with a MicroTik 10GbE Switch. I logged into the switch and saw that it had some how reverted to RouterOS again and not SwitchOS. Anyway, I switched it back to boot in the SwitchOS and then my verify speeds were higher that the writes as expected. I also switched the user account to "microsoft account' as the same time so don't know if it may actually be related to that. Unlikely but thought I'd throw it in there as things to try.

As I said, it could be totally unrelated but you never know :)

Cheers

The "microsoft account" checkbox only applies to SMB authentication. It's related to the default auth credentials sent to remote SMB servers.

If you're using a "microsoft account" to log into a Windows client (linked to an email address - foo@bar.com for example), then the client will send foo@bar.com over the wire as a username, and when it errors out, prompts the user for a username/password.

The way the "microsoft account" feature is intended to work is to map a local user account name on the FreeNAS server to its email address. So if you have a user foo with an email address foo@bar.com, then a mapping is generated foo@bar.com -> foo. This then allows the default Windows behavior to "magically" work.

In reality, this is significantly problematic approach for the following reasons:
1. It can introduce delays and potential bugs in authentication processing on the SMB side.
2. It requires keeping your email account's password in sync with the FreeNAS account password.
3. It requires keeping an NT hash of your email account's password.

For these reasons, I'll probably remove the MS account "feature" in a future version of TrueNAS.

 

[mjflower]

Ah ok, yeah not worth keeping then like you said if it just fails out anyway initally and then reverts.

Thanks for the info :)