How to troubleshoot or improve slow SMB transfers?

[ChrisNAS]

Hello,

I'm moving a bunch of files my system to a FN box, hundreds of gigs. However, dealing with anywhere from 1-20mbps is making this take way too long. I've read through several of the related threads here, and I'm not clear on how to troubleshoot the cause.

Running iperf doesn't seem to do anything... and that's probably a port or firewall thing. I'm not sure.
I just went out and purchased two new ethernet cables between the two systems and the router. No difference.

FN BOX:

TS-140
Build FreeNAS-11.0-U1 (aa82cc58d)
Platform Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
Memory 16138MB
Builtin gigabit ethernet
2x month old NAS seagate 4tb drives
FN on SSD
Jails on nvme

MY SYSTEM:

MSI MB builtin gigabit port
16gb ram
2x standard seagate 2tb drives
Windows 7 pro

ROUTER:

ASUS N66u gigabit ports

Some info I've seen requested:

Code:

root@freenas:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=40098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO>
        ether 6c:0b:84:ac:9a:52
        inet 192.168.2.10 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: iohyve-bridge
        ether 02:b5:85:a5:11:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: iohyve-ubuntu17
        options=80000<LINKSTATE>
        ether 00:bd:04:c6:f7:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
        groups: tap
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:e0:00:05:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Code:

root@freenas:~ # smbstatus

Samba version 4.6.4-GIT-5fe3bc2
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing

----------------------------------------------------------------------------------------------------------------------------------------
5487    Chris        TheGroup      192.168.2.2 (ipv4:192.168.2.2:49308)      SMB2_10           -                    -


Service      pid     Machine       Connected at                     Encryption   Signing
---------------------------------------------------------------------------------------------
Personal     5487    192.168.2.2   Sat Jul 15 12:17:13 2017 PDT     -            -

Locked files:
Pid          Uid        DenyMode   Access      R/W        Oplock           SharePath   Name   Time
--------------------------------------------------------------------------------------------------
5487         1001       DENY_NONE  0x100081    RDONLY     NONE             /mnt/Raid/Personal   Raid C/FN   Sat Jul 15 18:33:03 2017
5487         1001       DENY_NONE  0x100081    RDONLY     NONE             /mnt/Raid/Personal   Raid C/FN   Sat Jul 15 18:33:45 2017
5487         1001       DENY_NONE  0x100081    RDONLY     NONE             /mnt/Raid/Personal   Raid C/FN   Sat Jul 15 18:33:54 2017
5487         1001       DENY_WRITE 0x120089    RDONLY     LEASE(RWH)       /mnt/Raid/Personal   Raid C/FN/debug-freenas-20170715183204.tgz   Sat Jul 15 18:33:03 2017
5487         1001       DENY_NONE  0x100081    RDONLY     NONE             /mnt/Raid/Personal   .   Sat Jul 15 12:18:17 2017
5487         1001       DENY_ALL   0x100080    RDONLY     NONE             /mnt/Raid/Personal   .   Sat Jul 15 12:18:17 2017
5487         1001       DENY_NONE  0x100081    RDONLY     NONE             /mnt/Raid/Personal   Raid C   Sat Jul 15 17:01:45 2017

 

[tvsjr]

Start at layer 1 and work up. You say iperf doesn't work... assuming your FreeNAS and client system are on the same subnet, it should. If it doesn't, either you don't understand how to use it or something's broke.

You are doing this with a wired connection, yes?

You haven't listed the motherboard in your FreeNAS. If it has certain brands of NIC on-board, they may suck (Realtek).

Basically, your issue is "slow network performance", not "slow SMB performance", until you've proven it's actually an SMB issue.

 

[ChrisNAS]

Thanks for reply.

Onboard ethernet is an Intel I217 based. And yes, wired. Brand new cat5e between both systems and router. All gigabit.

I've seen someone talking about some options to change in smb conf on FN to speed things up, just not sure which or why yet or if apply to specifics. And I was looking into getting my win7 to use samba 3/4 instead of default 2x just not sure if worth the trouble of doing that.

 

[anodos]

I've seen someone talking about some options to change in smb conf on FN to speed things up

The options that people throw around are usually just voodoo. If there was a "go fast" parameter, then it'd be default already. ;-)

Are these large sequential files (movies) or small files? If they're mostly tiny files, what speeds do you get with a large sequential transfer?

 

[melloa]

Running iperf doesn't seem to do anything...

iperf measure network speed, so what you mean with doesn't seem to do anything?

That's what I get between one of my VMs and the FreeNAS box:

Code:

[root@centplex ~]# iperf -c 10.10.10.200
------------------------------------------------------------
Client connecting to 10.10.10.200, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.201 port 33952 connected with 10.10.10.200 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes  936 Mbits/sec

 

[Michal Smrž]

I would like to join this topic.
I also have not-so-performant Samba on Freenas 11.0-U2

HW:

• CPU E3-1241 v3 @ 3.50GHz
• 16 GB DDR3 ECC
• Supermicro X10SLA-F
• 3 x HGST 7K6000 in Adaptec 6805E : RaidZ1 (after testing 7 disks in RaidZ2)

Iperf 940->960MBoud

FTP, NFS going 100-110MB/s

Samba cupped at ~70MB/s, network interface ~800Mb/s, cpu load low (no core above 15%)
Without autotune, with autotune (some socket optimalisations), no change. DD write tests about 300MB/s

It's future company backup NAS and Samba is main protocol here, so it would be nice to have Samba faster.

 

[nightshade00013]

For the OP, I had something similar happen to me not too long ago. My UPS has a cat5 port on it and I was passing the ethernet through for protection. I made the change when I was shifting some stuff around and didn't even think about it. Anyway transferring files was abysmally slow, moved the cable and back to gigabit speeds. The UPS was only connecting at 100 Mbps speeds but it was showing the FreeNAS at gigabit connection.

@Michal Smrž I believe that is about the speed you will get with Samba on a gigabit connection. If something faster is needed you would need to step up to 10Gb networking.

 

[Michal Smrž]

And that's what brings me here to How to troubleshoot or improve slow SMB transfers?

If other protocols working fine and there is no cpu load, neither network interface cup, why, where...? It is realy just samba jumping into sleep, or some iowait? I read others get samba 110MB on gigabit. Well, not on Freenas and it's not super reliable source, so ...

Every user here have 70MB on gigabit and is fine with it?

 

[ChrisNAS]

Just wanted to update here...

So after checking router, drivers, settings, putting in new cables etc.., I was still in the same place. Slow transfers.

Well, I ended up having to build a new workstation to work on a large video project for a client, and with this new system I consistently get 100+ mb transfers with the same hard drives in both the new and the old system

Even though my old system had a gigabit card and all, it appears to me that the problem was with either of the older mb/cpu/network card/mem or very well could have been windows.

However, the new system is same OS... W7 Pro 64, just a fresh install on NVME (but data transfers still from orig drives. indicating that the drives were not the problem in old system)

So, in my case a new system solved the problem. I'm very happy with the new build, too! I highly recommend Ryzen.

 

[saikee]

Every user here have 70MB on gigabit and is fine with it?

I suppose I am one of them.

To me the math is simple. A gigabit network card does 1,000,000,000 bit per sec or 125,000,000 bytes per sec or 125MB/s max. That is a theoretical throughput as in transmission there must be handshakes and protocol involves. As a files is sent in packets so the smaller files consume much more overhead than large files. In practical term everyday usage the realistic average throughput is about half of the 125MB/s so 70 MB/s is normal to me when transferring files via a gigabit port and a network cable.

I am in the middle of transferring 15TB data and the first 3.9TB folder took 15.5 hours with an average of 70MB/s using Grsync in a 14 years old pc. The files were mainly videos of 1 to 3GB sizes from one FreeNAS to another FreeNAS box with a home network. I am using a third computer running Linux doing the file transfer by NFS shares.

I did have from time to time on small number of mainly large files writing speed over 90MB/s in an ordinary home network.

To increase the speed of file transfer the best way in do it internally between vdev. The consumer grade hdds I used are 8TB with quoted 200MB/s maximum write speed which I have managed without problem as I run a 3x2x8TB mirror source which read 6 disks to write on one output disk. I can't do it with the current setup which has a 5x8TB Raidz and my largest consumer grade box can accept only 10 hdd.

Thus in my case the limit of the hdd is 200MB/s achievable if I do the file transfer inside the same box. Outside the box I consider an average speed 70MB/s is not unreasonable.

 

[b7842]

I also met this problem.
Having changed all onboard lan to intel i350-t4 and i210, I found that Restarting the SMB service is the temp solution.
It can reach 11x MB again. Before restarting the service, it was around 77MB and sometimes 89MB. This problem was exist from 9.10.x to 11.x.x but the developers do not care about it.

 

[nojohnny101]

On large file types, transferring between my laptop and FN box, though hardwire, I usually see sustained speeds around 110MB/s and see peaks up to 115MB/s.

 

[biped]

I'll jump in here as well with mediocre smb performance ( although it is only transport method I'm using ).

getting around 60MB/s transferring large files to FN box.

On Fn11.1, iperf3 gets 940mbps one way and about 640mbps the other (-R parameter enabled on client). This tested from various sources with various amounts of copper and switches etc between. I suspect the crappy onboard realtek (re?) is probably not helping. The results are always the same.

Autotune is not enabled.

bozo@hostname:/mnt/dataVol02 # dd if=/dev/zero of=./test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 4.093350 secs (5123314671 bytes/sec)

bozo@hostname:/mnt/dataVol02 # dd of=/dev/null if=./test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 2.444883 secs (8577720387 bytes/sec)

I doubt those are useful for anything, given the results. Not even sure where I got the BS and Count parameters from.

Have been digging into the forums for a while now, but have to really pick and choose, there doesn't seem to be an established method of 'profiling' a system to help identify bottlenecks...

Seems to be 'run iperf' and check for network issues. All ok there ? Ok lets grab the shotgun and start fishing in a barrel :) While a nice roundabout way of absorbing insights from a lot of smart people, it is also frustrating.

 

[nojohnny101]

I suspect the crappy onboard realtek (re?) is probably not helping. The results are always the same.

That is the first thing I would change, it will cost you very little and will eliminate a wild card in troubleshooting that will always leave you thinking "but it could be..."

 

[biped]

Found an intel 'pro' nic, plopped it in. Device em0 now happily directing traffic. iperf3 is now symmetric, as nature planed.

SMB transfer still less than arousing.

Seems like issue with 'signing' feature on some recent versions of osx. Disabled ... and ...

SMB transfer still less than arousing.

Time to find more shotgun shells.

 

[biped]

On a whim, decided to profile the local 'source' disk I was transferring the large files from, that didn't look to good to my surprise. Moved the files to a local SSD, and tested again, and ...

SMB transfer now very arousing !

Well, I hope this helps the others who might fall upon this thread.