Yet another poor CIFS performance thread

Status
Not open for further replies.

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
This thread has grown out of my build thread in the Hardware section. I will be using this post to display testing results. Please speak up if you see anything that might help!

After building my new FreeNAS box (specs in my signature) I am getting poor CIFS performance. Large transfers start out at 100MB/s and quickly drop to 50MB/s.

First off, some things about configuration and overall setup:
- The FreeNAS motherboard has dual Intel NICs.
- The desktop PC used for testing also has an Intel NIC.
- The switch connecting all of the devices is a Netgear GS108E.
- I have enabled autotune which has added the following settings:
kern.ipc.maxsockbuf= 2097152
net.inet.tcp.recvbuf_max= 2097152
net.inet.tcp.sendbuf_max= 2097152
vfs.zfs.arc_max
vm.kmem_size
vm.kmem_size_max
- I have tested disabling AIO to no effect (it was previously enabled).
- I have increased the MTU to 9000 on the FreeNAS NIC.

Some test results:
- DD tests
Code:
[root@freenas] /mnt/volume# dd if=/dev/zero of=testfile bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.297915 secs (171861112 bytes/sec)
[root@freenas] /mnt/volume# dd if=testfile of=/dev/zero bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.122954 secs (416415455 bytes/sec)


- Iperf tests
With default settings I get a consistent 52MB/s between server and client. When I modify the TCP window size in Jperf it instantly jumps to 112MB/s.

I have noticed that during file transfer tests the network utilization on the desktop machine continually spikes up into the 80 - 90% utilization range and then down to 0% and then back up again. When running the Iperf tests the network utilization stays at 99%.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I tested the same file transfer on a VM that has a physical drive mapped to it and I received much better performance; approximately 80 - 100 MB/s. However I still saw the same network utilization spikes on the client machine despite the higher total throughput.
I should note that this VM lives on my ESXi host so it is a completely different machine. The ESXi host also uses an Intel NIC.

Writing to the FreeNAS box from that same VM yields about 60MB/s+ performance but the network utilization is much more consistent, bouncing between 50% and 65% rather than 0% and 90% on my desktop machine for reads. Even transferring to the ESXi VM i noticed a couple of drops down to 0%. I then found this:

Code:
[root@freenas] /# dmesg |grep em1
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xe000-0xe01f mem 0xdf900000-0xdf91ffff,0xdf920000-0xdf923fff irq 16 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: Ethernet address: ************ [my edit]
em1: link state changed to DOWN
em1: link state changed to UP
em1: link state changed to DOWN
em1: link state changed to UP


Just to make sure I restarted the server and ran the file transfer again but this time there was nothing in dmesg about em1 despite the same test results.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I highly recommend disabling your jumbo packets. Unless every network device,network switch, etc. supports an MTU of 9000 and has it enabled you will have weird network problems that manifest themselves with issues ranging from network disconnects to poor performance. Most home routers, network printers, most home network switches, wifi hubs, etc. do not support an MTU of 9000 so you almost certainly have at least 1 device that doesn't support jumbo frames. I couldn't find anything that said that the GS108E supports jumbo frames, so I'd bet money it doesn't support it. Manufacturers love promoting jumbo frames because it's another feature that many uninformed people think will increase network performance. The truth is that with such a low latency as is found on a home network jumbo frames doesn't really help.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
Jumbo frames have already been disabled. I am making a change, testing it, and then reverting to original settings.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I'm baffled. This is getting pretty annoying.

Are there any other tests I can run that might shed some light on what is going on?

I enabled FTP to do some testing and received the same results as CIFS: about 60MB/s
 

paleoN

Wizard
Joined
Apr 22, 2012
Messages
1,403
vfs.zfs.arc_max
vm.kmem_size
vm.kmem_size_max
Are these unset? Any and all tunables/sysctls if you please.

- DD tests
Those tests are worthless. Read the 1[sup]st[/sup] post of the [thread=981]performance sticky[/thread] to see why. In another session you can run:
Code:
gstat -I1s -f 'a?da[0-9]+$'
Check for any lagging disks.

I enabled FTP to do some testing and received the same results as CIFS: about 60MB/s
To be clear this is on reading? I assume the client is capable of keeping up?
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
vfs.zfs.arc_max=10596385843
vm.kmem_size=11773762048
vm.kmem_size_max=14717202560

I have also tried vfs.zfs.prefetch_disable with values of both 1 and 0 - both provide the same results.

Yes, this is for reads and the client is able to keep up. All of my computers are based on Intel sock 1155 CPU's and have Intel NICs.

Using the command above gives me a nice little table in my ssh session. When I perform the file transfer test (read via CIFS) it shows the disks being used and then dropping to 0% and then back up again.
 

paleoN

Wizard
Joined
Apr 22, 2012
Messages
1,403
vfs.zfs.arc_max=10596385843
vm.kmem_size=11773762048
vm.kmem_size_max=14717202560
You can try deleting both kmem_size tunables and changing vfs.zfs.arc_max to 14G. Not that I think this is related to your current problem.

The client is able to keep up. All of my computers are based on Intel sock 1155 CPU's and have Intel NICs.
How about the storage subsystem of the client? You already eliminated the network as far as I'm concerned.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I will try running the following commands and post the results:

Code:
dd if=/dev/zero of=tmp.dat bs=2048k count=50k

Code:
dd if=tmp.dat of=/dev/null bs=2048k count=50k
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
Results:

Code:
[root@freenas] /mnt/volume# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 217.642786 secs (493350523 bytes/sec)
[root@freenas] /mnt/volume# dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 192.630605 secs (557409776 bytes/sec)


While performing the read test 4 of the 6 drives normally stayed between 40 - 50% busy while the other 2 drives stayed around 90% busy as measured by the command you provided above. This seems odd, but I'm really not sure.
 

paleoN

Wizard
Joined
Apr 22, 2012
Messages
1,403
While performing the read test 4 of the 6 drives normally stayed between 40 - 50% busy while the other 2 drives stayed around 90% busy as measured by the command you provided above. This seems odd, but I'm really not sure.
It does seem a bit odd. Still the pool performed quite well.

These network sysctls are likely a bit higher than needed, but...:
Code:
kern.ipc.maxsockbuf = 16777216
net.inet.tcp.recvbuf_max = 16777216		
net.inet.tcp.sendbuf_max = 16777216


CIFS:
Code:
socket options = TCP_NODELAY SO_KEEPALIVE
write cache size = 131072
I forget exactly what I have, but it's something like that.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
My CIFS Aux parms:
Code:
write cache size = 131072
socket options = TCP_NODELAY SO_SNDBUF=524288 SO_RCVBUF=524288


I added SO_KEEPALIVE with the same results. The network utilization graph on my client still looks like this: /\/\/\/\/\/\/
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
Using the gstat command above reveals the following, which may or may not be useful in some way:

Test 1:
1x 100GB file copied to desktop machine from FreeNAS box. Average throughput is 65 - 70 MB/s.
Network utilization on client machine bounces around a bit but stays between 50 - 60% and doesn't drop to 0%.
Code:
dT: 1.001s  w: 1.000s  filter: a?da[0-9]+$
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0     71     71   8216    4.9      0      0    0.0    4.4| ada0
    0     71     71   8216    5.7      0      0    0.0    5.5| ada1
    0     72     72   8152    7.0      0      0    0.0    5.8| ada2
    0    136    136  16375    5.1      0      0    0.0    7.7| ada3
    0     72     72   8152    7.1      0      0    0.0    5.9| ada4
    0    136    136  16375    5.0      0      0    0.0    7.6| ada5
    0      0      0      0    0.0      0      0    0.0    0.0| da0

Disk % busy stays reasonably constant.

Test 2:
1x Folder containing 27 files @ 231MB each for a total of 5.96GB. Average throughput is 50 - 55 MB/s.
Network utilization on client machine looks like /\/\/\/\/\
Code:
[root@freenas] ~# gstat -I1s -f 'a?da[0-9]+$'
dT: 1.001s  w: 1.000s  filter: a?da[0-9]+$
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0    154    154  16883    3.1      0      0    0.0    6.4| ada0
    0    156    156  16899    4.9      0      0    0.0   13.6| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0    154    154  16883    3.3      0      0    0.0    6.8| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ada4
    0    157    157  16899    4.0      0      0    0.0   10.0| ada5
    0      0      0      0    0.0      0      0    0.0    0.0| da0


When the network utilization drops to 0% on the client machine the disk % busy on the FreeNAS box also drops to 0%.


Edit: I performed these same tests on another machine (VM) and the throughput was improved somewhat but still not acceptable given the build. I used an old hard drive in my current desktop build and am wondering if write performance has degraded. I will swap in a new WD blue 1TB and after I get windows installed I'll post more results. In the mean time I'd love some more suggestions. Thanks for the help so far folks!
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
As part of my build I bought a new USB stick and put the latest version of FreeNAS on it. I still have the old USB stick with a previous version on it so I'm wondering if I would be able to swap the USB sticks to test the old version. Would that hurt anything?
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I put a new hard drive in my windows machine and got everything running again which actually yielded pretty good results. I went from 50 MB/s to 70MB/s or so. I'm still thinking that I might be limited by this (new) hard drive because the write to the FreeNAS box went along nicely at 100MB/s. I will do some more testing but it definitely seems like the write performance wasn't where it should have been on that other hard drive.

I'll do some performance benchmarking on this new hard drive to see what I can expect in terms of write speed.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
In an effort to eliminate external disk performance issues I created a 4GB RAM disk on the client machine. Reads from the NAS hovered around 85MB/s and writes to the NAS never dipped below 100MB/s. Interesting results.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I removed all of the Aux parameters from the CIFS configuration and am now seeing a consistent 100MB/s+ in both directions to the RAM disk. Seems like the client disk performance was the issue. Is this a reasonable conclusion?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It is. I never do any speed tests to spinnng rust because most of them can't achieve the required speeds to really "test" the performance of Gb. When you mentioned that you had switched hard drives I kind of wondered why you were using spinning disks to test performance. They're a very poor test because a small amount of fragmentation can kill the throughput thanks to relatively high seek times compared to an SSD.
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
This is the first NAS box I have ever built so the process of setting it up, configuring it, and testing it is all new. It has been a good learning experience though. I'll know for next time.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Anytime I do speed tests I use an SSD or a RAMdisk. A free open source RAMDISK option is called imdisk. It's not bloated and you can create and destroy your ramdrive in seconds.
 
Status
Not open for further replies.
Top