Performance difference between iSCSI and CIFS robocopying 17GB music file archive

[sysfu]

I recently setup a FreeNAS iSCSI disk to use with a Windows 7 workstation. My immediate perception was that is was faster than working with the same files via a SMB/CIFS share.

So I ran some tests. The first one is writing the entire 17GB archive of music files to the FreeNAS box via SMB/CIFS and then via iSCSI. CIFS took ~22min, iSCSI ~17min.

The second test is a robocopy operation verifying the entire file tree. iSCSI blows away CIFS performance on that one. CIFS = ~69sec iSCSI = ~4sec

PS C:\> Measure-Command {start-process robocopy -argumentlist "t:\music \\nas\files\music /mir" -wait}


Days : 0
Hours : 0
Minutes : 21
Seconds : 48
Milliseconds : 563
Ticks : 13085630799
TotalDays : 0.0151454060173611
TotalHours : 0.363489744416667
TotalMinutes : 21.809384665
TotalSeconds : 1308.5630799
TotalMilliseconds : 1308563.0799



PS C:\> Measure-Command {start-process robocopy -argumentlist "t:\music i:\music /mir" -wait}


Days : 0
Hours : 0
Minutes : 16
Seconds : 51
Milliseconds : 982
Ticks : 10119829731
TotalDays : 0.0117127658923611
TotalHours : 0.281106381416667
TotalMinutes : 16.866382885
TotalSeconds : 1011.9829731
TotalMilliseconds : 1011982.9731



PS C:\> Measure-Command {start-process robocopy -argumentlist "t:\music \\nas\files\music /mir" -wait}


Days : 0
Hours : 0
Minutes : 1
Seconds : 9
Milliseconds : 301
Ticks : 693012306
TotalDays : 0.000802097576388889
TotalHours : 0.0192503418333333
TotalMinutes : 1.15502051
TotalSeconds : 69.3012306
TotalMilliseconds : 69301.2306



PS C:\> Measure-Command {start-process robocopy -argumentlist "t:\music i:\music /mir" -wait}


Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 889
Ticks : 38897811
TotalDays : 4.50206145833333E-05
TotalHours : 0.00108049475
TotalMinutes : 0.064829685
TotalSeconds : 3.8897811
TotalMilliseconds : 3889.7811

 

[Daisuke]

Thanks for the tests, please post your steps to setup iSCSI in FreeNAS and Windows.
I tried this tutorial and I cannot start the iSCSI service on FreeNAS 8.3, once all params are configured.

 

[sysfu]

I used the same tutorial to get my setup going and it worked like a champ. Have you logged into your FreeNAS system via SSH and checked /var/logs for clues as to why the iSCSI service will not start?

 

[jgreco]

iSCSI works fine on FreeNAS 8.3.0, but you do have to walk through all the steps and (my gripe!) they're in kind of a logical order, but not an order that you can follow straight down to make it all work.

 

[Daisuke]

I used the same tutorial to get my setup going and it worked like a champ. Have you logged into your FreeNAS system via SSH and checked /var/logs for clues as to why the iSCSI service will not start?

I get this in /var/log/messages:

Code:

Dec  1 18:25:24 pluto notifier: istgt_lu.c:1674:istgt_lu_add_unit: ***ERROR*** LU1: LUN0: Auto size error (/mnt/nas/media/extent)
Dec  1 18:25:24 pluto istgt[95540]: istgt_lu.c:1674:istgt_lu_add_unit: ***ERROR*** LU1: LUN0: Auto size error (/mnt/nas/media/extent)
Dec  1 18:25:24 pluto istgt[95540]: istgt_lu.c:1996:istgt_lu_init: ***ERROR*** lu_add_unit() failed
Dec  1 18:25:24 pluto notifier: istgt_lu.c:1996:istgt_lu_init: ***ERROR*** lu_add_unit() failed
Dec  1 18:25:24 pluto istgt[95540]: istgt.c:1669:main: ***ERROR*** istgt_lu_init() failed
Dec  1 18:25:24 pluto notifier: istgt.c:1669:main: ***ERROR*** istgt_lu_init() failed
Dec  1 18:25:24 pluto root: /usr/local/etc/rc.d/istgt: WARNING: failed to start istgt
Dec  1 18:25:24 pluto notifier: /usr/local/etc/rc.d/istgt: WARNING: failed to start istgt

/mnt/nas/media/extent is a 0kb file, I have no disks available as they are all used on my RadZ2 array. I even tried a different location, same error as above:

Code:

# touch /mnt/nas/extent
# ls -lha /mnt/nas
total 198
drwxr-xr-x   5 root    wheel        6B Dec  1 18:32 .
drwxr-xr-x   4 root    wheel      512B Oct 27 17:14 ..
drwxr-xr-x   7 root    1001         8B Oct 26 19:03 .freenas
-rw-rw-r--   1 root    wheel        0B Dec  1 18:32 extent
drwxr-xr-x+ 10 floren  managers    18B Nov 25 00:59 media
drwxr-xr-x   7 root    wheel        7B Feb  6  2012 unix

jgreco or anyone, can you post the proper steps here?
I have CIFS, NFS and SSH services enabled.

 

[jgreco]

How big do you want the extent to be?

 

[Daisuke]

Do I have to define it? I thought leaving it at 0 it will take whatever space is available in a dynamic way.

 

[jgreco]

No. You can't do that with ZFS. ZFS is a copy-on-write filesystem, which means that new blocks are written somewhere other than the old location, so if you were able to let it take all the space and it actually did so, your filesystem would grind to a halt. ZFS filesystems degrade performance-wise past 80% full in any case. For iSCSI use, I'd actually suggest that past 60% might be bad...

Anyways, you've told it a zero-byte extent. Apparently istgt won't serve that.

You can create your extents one of two ways.

The quick and stupid: "truncate -s 500G extentfilename"

The upside to this is that it's quick. The stupid part is that it doesn't allocate the space right away, but rather as it's used, and eventually you'll do something dumb and suddenly run out of space. So don't do this.

The slow and reliable: "dd if=/dev/zero of=extentfilename bs=1048576 count=***x" where ***x is how many megabytes you want it to be.

 

[Daisuke]

OK, defining a specific extent size does fix the issue, so I can start the service and configure the iSCSI disk in Windows 8. However, the performance is not fantastic for me. I formatted the disk in Windows first as basic then as dynamic, hoping it will make a difference in performance. It does not. While transfering a 10GB file, the transfer rate stays at 115-120MB/sec for the first 20 seconds then it drops quickly fluctuating to values from 3MB/sec (yes 3) to 50MB/sec. At least with CIFS it fluctuates between 50MB/sec and 60MB/sec.

 

[jgreco]

Okay, good. Now go read bug 1531.

 

[Daisuke]

Looks like something inherited from FreeBSD which I doubt it will be touched by devs...
The bug report was open 7 months ago, with no resolution so far. It sucks because for for the first 20 seconds when I ran the iSCSI test, I was drooling at the 120MB/sec transfer.

 

[jgreco]

Wow, it's really been a long night, I was sure I had already responded to this.

This isn't a FreeBSD problem, really. It's a ZFS thing, and it combines poorly with design decisions made by the FreeNAS team that were meant to address *other* performance issues. The FreeNAS guys decided to optimize FreeNAS to work in a "reasonable" amount of memory ("reasonable" for ZFS, insane most other places) and that ends up being 6-8GB for smaller systems, in order to get a reasonably sized ARC and other stuff. I can at least understand the strategy, because if they had said "ok guys the minimum is 1GB", they'd forever be tracing down performance issues that are basically a side effect of ZFS not having the memory it wants. But by giving it lots of memory, that /can/ introduce other problems. I spent a lot of time characterizing the issue with multiple machines, multiple vdev layouts, and lots of tuning tweaks.

While the bug may not be "resolved" in that a comprehensive fix isn't available (and probably won't be, at least not the way we usually think of a fix), there is more than enough information there for you to "fix" your system. The trick is to characterize what sort of performance you can expect out of your pool under stressy conditions. My benchmark for that was simultaneously reading a large file, writing a large file, and scrubbing the pool. I considered performance acceptable when the system was responsive under that load. There's ALWAYS a server-melting load, you just want the melting point to be bigger than what can be thrown at the server over the network.

So until the FreeNAS guys come up with a way to work some load testing into autotune or something like that, the resolution is to do what I outlined in 1531, which is why I gave specific steps to take. My guess? You might not end up with a 120MB/sec system, but you'll get a system that's pretty fast.

 

[Daisuke]

OK jgreco, I'm all years. I really don't expect to get 120MB/sec but if I will get more than 60MB/sec (constant, not variable like I explained before), I will be very happy... as this is the speed I get now with CIFS. Why don't you outline in this thread a condensed set of commands for all of us, instead of posting a lot of info in your ticket? Thanks :)

 

[jgreco]

Because you really need to understand what I did and why I did it in order to have a reasonable hope of reproducing the results? If it was just a matter of setting sysctl vfs.zfs.makeitnotsuck.ouch.ouch.ouch=1 I'd tell you.

 

[cyberjock]

If it was just a matter of setting sysctl vfs.zfs.makeitnotsuck.ouch.ouch.ouch=1...

LOL!

I tried that sysctl and it didn't help. What else can I try? :P

 

[jgreco]

The Swedish Chef fun is still over in alt.swedish.chef.bork.bork.bork ... they'll have some ideas I'm sure.

 

[Daisuke]

Because you really need to understand what I did and why I did it in order to have a reasonable hope of reproducing the results? If it was just a matter of setting sysctl vfs.zfs.makeitnotsuck.ouch.ouch.ouch=1 I'd tell you.

Are you serious? We are all technical people here on these forums, not brainless copy/pasters.
Why don't you post something concise and effective, with a real example.... instead of more phraseology? "I performed the following tests with this configuration on my hardware" is a lot more relevant, don't you think?

 

[cyberjock]

In jgreco's defense I think what he's saying is that it's not a matter of he doesn't want to help you. It's that the tunables really need to be created by you based on your usage pattern, your hardware, etc. It's not easy for someone to just say "add these settings to improve performance". It's a matter of knowing how your system works and figuring it out based on lots of testing.

I explain to people that tweak stuff and then get upset when the system doesn't work right like this: There's a reason why the default is the default. There is a tradeoff in everything you do, in computing, in life, etc. If there was no downside that would be the "default". So it's important to analyze your setup and figure out what the best "default" is for your exact situation.

 

[Daisuke]

Spoonfeed me? No thanks. Anyways, your above post is a "nice condensed" set of phrases having as end result more wasted time. The fact that you spent all this time to quote and format the reply (while going again in circles) says it all to me. You act like you are making someone a favor by sharing some drops of God given knowledge with your pipette. Rest assured, I'll live very well with my current setup and without your arrogant posts. I've been working with Linux for over 15 so I managed to see some of the "Gods" like you. As a side note, I will not respond to any of your ridiculous posts because you are on the Ignore list.

 

[jgreco]

It never ceases to amaze me, this attitude by some people who feel it is perfectly acceptable to demand that someone else do their homework for them.

Boggles the mind.