Making copy of file in same NAS directory SLOW

Jerizo · Oct 25, 2015

I am having some trouble with Freenas performance. Making a copy of a 350MB File through CIFS in the same network directory results in 3.16MB/s.
Also copying files from my workstation over to the NAS only runs at a rate of 6MB/s write and 18MB/s read.
Everything is hooked up through gigabit switch. In case this matters - the zfs Volume is a Mirror of 2 HDDs, WDVelociraptor 10k rpm and SamsungSpinpoint 7.2k rpm (this only a testing setup, WDReds are ordered).

So there definitely is something wrong with my setup in terms of performance, generally speaking, but the fact that making a file copy on the NAS is that slow and even slower than transferring a file from workstation to NAS, leaves me clueless.

ifconfig output:

Code:

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether d0:50:99:83:46:88
        inet 192.168.178.28 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

system specs:

Code:

NAS Box:                                 Workstation:
OS:         Freenas 9.3                 OS:         Windows 7
BOARD:      ASRock N3700-ITX            BOARD:      Asus x99-Deluxe
CPU:        Intel Pentium N3700         CPU:        Intel i7-5960X
NIC:        Realtek L8111GR (onboard)   NIC:        Intel I218V (onboard)
RAM:        16GB  (NON-ECC)             RAM:        32GB

Network Hardware:
Switch: Netgear GS105 (5-Port Gigabit)
Router: AVM FRITZ!Box 3490

Network Infrastructure:
[Router]->[Switch]->[Nas Box, Workstation]

DifferentStrokes · Oct 25, 2015

The Realtek NIC isn't helping things. They are know to have weird problems in BSDland. I'd start by picking up a Intel NIC.

jgreco · Oct 25, 2015

Please use "iperf" to check your network. As @DifferentStrokes noted, the Realtek interfaces are powered by either one or two hamsters running on an exercise wheel to move your bits around. We find that often one of them gets sick or maybe dies; this usually has a severe negative impact on performance. You can probably find dozens of other examples if you Google "+hamster site:forums.freenas.org"

Picking up an inexpensive desktop-grade Intel CT ethernet adapter is the best fix for that particular problem. However, the Realtek ethernets are usually a sign that you're also using other non-recommended hardware, so I'd ask that you please post your full system inventory. You'll kinda get beat up a little if it's a $50 desktop board; take it in good humor because ultimately we just want you to have a high performance, reliable system and the guys here will definitely help get you sorted out if you let them.

Jerizo · Oct 25, 2015

First of all thank ya’ll for the crazy fast help!

@jgreco I’m not sure what other system specs you would be looking for, I thought I had posted all the relevant data.
I should add however, that I am using non ECC memory which I am aware is definitely not recommended. The research on the risks I did before hand, did allow me though, to justify this decision with budget considerations in mind.

I will order an Intel Gigabit CT Desktop Adapter today and see if that helps.

Here are the iperf numbers:

Command executed on NAS:

iperf -s -i 5

Command executed on Workstation:

iperf -c 192.168.178.28 -t 30 -d

Code:

Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 7] local 192.168.178.28 port 5001 connected with 192.168.178.24 port 3718
------------------------------------------------------------
Client connecting to 192.168.178.24, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 9] local 192.168.178.28 port 59409 connected with 192.168.178.24 port 5001
[ ID] Interval Transfer Bandwidth
[ 7] 0.0- 5.0 sec 7.77 MBytes 13.0 Mbits/sec
[ 9] 0.0- 5.0 sec 29.4 MBytes 49.3 Mbits/sec
[ 7] 5.0-10.0 sec 7.81 MBytes 13.1 Mbits/sec
[ 9] 5.0-10.0 sec 30.4 MBytes 51.0 Mbits/sec
[ 7] 10.0-15.0 sec 7.81 MBytes 13.1 Mbits/sec
[ 9] 10.0-15.0 sec 30.4 MBytes 51.0 Mbits/sec
[ 7] 15.0-20.0 sec 7.81 MBytes 13.1 Mbits/sec
[ 9] 15.0-20.0 sec 30.4 MBytes 51.0 Mbits/sec
[ 7] 20.0-25.0 sec 7.81 MBytes 13.1 Mbits/sec
[ 9] 20.0-25.0 sec 30.4 MBytes 51.0 Mbits/sec
[ 7] 25.0-30.0 sec 7.81 MBytes 13.1 Mbits/sec
[ 7] 0.0-30.0 sec 46.8 MBytes 13.1 Mbits/sec
Waiting for server threads to complete. Interrupt again to force quit.
[ 9] 25.0-30.0 sec 30.4 MBytes 51.0 Mbits/sec
[ 9] 0.0-30.0 sec 181 MBytes 50.7 Mbits/sec

Now there are some 50Mbits/s numbers in there, I had not observed those before. I wonder how I would be able to replicate that through CIFS.

jgreco · Oct 25, 2015

Jerizo said:
I should add however, that I am using non ECC memory which I am aware is definitely not recommended. The research on the risks I did before hand, did allow me though, to justify this decision with budget considerations in mind.

Only if it is recycled memory. The cost differential for new memory between non-ECC and ECC is usually modest.

Yeah, but it should be knocking out an easy 900+Mbits/sec in both directions. You have network issues.

One of the things to understand about NAS is that you are layering complex subsystems on top of other subsystems. Each layer introduces a further reduction of performance. If the iperf above were as good as it could ever get, you would NEVER be able to see 50Mbit/s of CIFS traffic, because all the other layers introduce additional latency and delays. The only way we get performance out of these things is to find the weakest links in each subsystem and to make them as small as possible. You could definitely get 30, maybe 40, possibly 45Mbps (with a tailwind) once all the other layers get through with it all.

Okay, I just finished running some iperf on two servers here. Now, I should note, these are both virtualized boxes, so that means we've got hypervisor vSwitches involved, virtualization involved, and traversal of at least two core switches here, and there's other production traffic on those hypervisors consuming some network capacity, but I got over 900Mbps in both directions.

Code:

% iperf -w 256k -c 10.249.2.141 -t 60; iperf -w 256k -c 10.249.2.141 -t 60 -R
------------------------------------------------------------
Client connecting to 10.249.2.141, TCP port 5001
TCP window size:  257 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.249.2.140 port 42115 connected with 10.249.2.141 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.51 GBytes   932 Mbits/sec
------------------------------------------------------------
Client connecting to 10.249.2.141, TCP port 5001
TCP window size:  257 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.249.2.140 port 12606 connected with 10.249.2.141 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.49 GBytes   930 Mbits/sec

See, and I would consider that non-stellar performance if it was on a physical platform. So, step one, go fix your network. Once you can get close-to-gigabit speeds out of iperf, then we can look at any other issues that pop up.

Jerizo · Oct 25, 2015

Got it! Intel NIC should arrive till Wednesday next week. I will make sure to update asap.
Thank you for elaborating. I feel like you help the Freenas community a great deal, read tons of your posts while researching.
Rest assured, your effort is most definitely appreciated.

p.s. the memory was recycled indeed, from my previous workstation.

jgreco · Oct 25, 2015

Oh and by the way I think where I was going with the system inventory request was probably that I wanted to know what kind of gigabit switch you had, and possibly the router (which we can agree SHOULDN'T be an issue). When looking to resolve network issues, it is best to look at everything.

Jerizo · Oct 25, 2015

Sure, I'm happy to provide as much info as possible. The Switch is a Netgear GS105 and the router is a AVM FRITZ!Box 3490.
Original post is now updated with this info.

jgreco · Oct 25, 2015

If you want to waste a half hour, try manually configuring IP's on both the NAS and the PC, disconnect the switch from both, and hook the NAS directly to the PC. See what kind of iperf results you get then.

Jerizo · Oct 29, 2015

Alight, I did not get around to trying a direct connection, I do have the results for new Intel NIC however.

iperf sending to nas:

Code:

[ 6] local 192.168.178.44 port 50851 connected with 192.168.178.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0- 5.0 sec 213 MBytes 357 Mbits/sec
[ 6] 5.0-10.0 sec 216 MBytes 362 Mbits/sec
[ 6] 10.0-15.0 sec 209 MBytes 351 Mbits/sec
[ 6] 15.0-20.0 sec 209 MBytes 350 Mbits/sec
[ 6] 20.0-25.0 sec 219 MBytes 367 Mbits/sec
[ 6] 25.0-30.0 sec 213 MBytes 358 Mbits/sec
[ 6] 0.0-30.0 sec 1.25 GBytes 357 Mbits/sec

iperf sending to workstation:

Code:

[ 6] local 192.168.178.23 port 46440 connected with 192.168.178.44 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0- 5.0 sec 465 MBytes 780 Mbits/sec
[ 6] 5.0-10.0 sec 462 MBytes 775 Mbits/sec
[ 6] 10.0-15.0 sec 463 MBytes 777 Mbits/sec
[ 6] 15.0-20.0 sec 461 MBytes 773 Mbits/sec
[ 6] 20.0-25.0 sec 462 MBytes 775 Mbits/sec
[ 6] 25.0-30.0 sec 460 MBytes 772 Mbits/sec
[ 6] 0.0-30.0 sec 2.71 GBytes 775 Mbits/sec

While this is a massive improvement, there still seems to be some headroom fore improvement. Uploading files to NAS through CIFS yields a transfer speed of about 70MB/s, while downloading from NAS runs at about 110MB/s. Making a copy of a file into the same NAS directory through CIFS works at speed of about 55MB/s.

jgreco · Oct 30, 2015

Between two Intel ethernets, I'd call that "probably broken." Definitely move on to removing the switch and trying a direct connection, also use the opportunity to check your cables by making sure they both run at full speed between the two machines.

Jerizo · Oct 31, 2015

I do remember having problems with my workstation NIC before. It would disconnect randomly, but those issues seemed to have been fixed by software updates.
Currently very busy, I might be able to try direct connection tomorrow. Will post results asap.

jgreco · Oct 31, 2015

Yeah, I definitely appreciate that it's inconvenient and annoying to have to mess around like this, but it is very helpful to try to isolate a specific item as being problematic... we should at least be able to isolate where the problem is.

Jerizo · Nov 1, 2015

Okay so I tried a direct connection between NAS and Workstation. The result was a set of almost the exact same numbers, which is why I will spare posting them.
I did notice however, that CIFS performance for moving data to the NAS increased from 70-90MB/s to a pretty stable 100MB/s. Download speed remained the same at 110MB/s.

Not sure how to interpret this.

Important Announcement for the TrueNAS Community.

Making copy of file in same NAS directory SLOW

Jerizo

Dabbler

DifferentStrokes

Patron

jgreco

Resident Grinch

Jerizo

Dabbler

jgreco

Resident Grinch

Jerizo

Dabbler

jgreco

Resident Grinch

Jerizo

Dabbler

jgreco

Resident Grinch

Jerizo

Dabbler

jgreco

Resident Grinch

Jerizo

Dabbler

jgreco

Resident Grinch

Jerizo

Dabbler

Similar threads