Network Performance?

[Hexland]

I posted my disk/system performance yesterday in the performance thread --
http://forums.freenas.org/showthrea...amarks-and-Cache&p=24532&viewfull=1#post24532

Last night, I played around with trying to transfer data to and from FreeNAS via CIFS, NFS and iSCSI.
Ideally, I want to use iSCSI (because FreeNAS will ultimately be a NAS for a Windows Home Server machine)... but I thought I'd experiment to see what performance will be like for each of the different network protocols.

Code:

ZFS, All drives formatted 4096K (gnop)
ZFS-RAID-2, 6x2TB (gnop)
-----------------------------------------------------------------

WRITE -- dd if=/dev/zero of=/mnt/test/test.dat bs=2048k count=50k
107374182400 bytes transferred in 568.901245 secs (188739581 bytes/sec) -- 179.99Mb/s

READ -- dd if=/mnt/test/test.dat of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 319.719823 secs (335838364 bytes/sec) -- 320.28Mb/s

CIFS -- TO NAS (51Mb/s) peaky (reaching 80-90Mb/s)
CIFS -- FRM NAS (67Mb/s) peaky (reaching 102Mb/s)

iSCSI (MB) -- TO NAS (81Mb/s) peaky
iSCSI (MB) -- FRM NAS (42Mb/s) hella peaky

iSCSI (Intel) -- TO NAS (71Mb/s) peaky
iSCSI (Intel) -- FRM NAS (49Mb/s) hella peaky

The connecting machine is a Core i7-930 with 16Gb RAM and 128Gb SSD and integrated Gigabit NIC.

A few things surprised me...

1. WRITE speed to the NAS was faster than reading from it
2. The integrated on-board NIC on the HP Microserver was actually faster than the Intel PCIe Gigabit Nic
3. Network throughput was extremely 'peaky' -- it could not sustain a consistent speed without dropping right down to 2-3Mb/s for a second or two before bouncing back up...
4. Although the average speed of CIFS was ultimately slower than iSCSI -- I was surprised to see that the upper speed range of CIFS was considerably higher than iSCSI...

Why is my write speed (transferring TO the NAS) faster than my read speed (reading FROM the NAS) -- this doesn't make sense to me, since the disk read speed is significantly faster than writing (and write operations are generally more expensive)?

Why does CIFS appear to be faster (although slower on average) during transfers -- I was under the impression that CIFS had a high protocol overhead compared to something like iSCSI?

Why are my network transfer speeds so peaky/bouncy?

Are there any guides to properly tune network performance, and perhaps make the speeds more consistent?

Are these numbers around the range to expect for this level of hardware?

Is ZFS-Raid-Z2 so CPU intensive that it is adversely affecting network performance?


Thanks guys, any help would be muchly appreciated..

 

[b1ghen]

Do you use jumbo frames?

 

[Hexland]

Not in this case, no.

 

[b1ghen]

I have had similar results with transfers being extremely peaky like you say, turning on jumbo frames made some improvements in my case but speed is far from stable.
What NIC do you have in your connecting machine, it says integrated but what is it?

 

[Hexland]

It's an Intel based motherboard in the i7 rig -- so it's most likely Intel NIC (I'll take a look when I get home)

 

[sunflashx]

In my experience the really peaky performance comes from not altering some of the default ZFS settings.

This is what I have in my loader.conf for a 8 gig system. The memory values were generated by the ZFS tuning tool that FreeNAS .7 Legacy has.

# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.zil_disable="1"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="4"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"