Disappointing performance

[Idiotzoo]

I'm building a production storage server for the charity I work for, where there's never the budget to buy quite the right tool for the job. I've used Freenas to store backups for years and I like it.

Hardware I'm currently playing with is:
AMD A4-5300 CPU on an A85 chipset board (GIGABYTE GA-F2A85X-D3H)
16GB ram
4 x Seagate Barracuda 3TB disks
HP NC380T dual NIC

I've configured ZFS with a pool of two mirrors, and I've tried RAID-Z1 and Z2 but the mirrored pairs offers the best performance.

However... That performance isn't great. Using DD I seem to be getting about 320MB/s write and 500MB/s read, which seems fine.

Over the network it's rarely maxing 1Gb and can drop alarmingly. Particularly if I copy some smaller files, when the speed can drop to 20KB/s

I'm seeing similar performance from iSCSI and CIFS.

I keep finding references to performance tweaks, but not the tweaks themselves.

Hopefully someone can help me.

 

[Stephens]

Some ideas...

- Show your actual dd commands used for testing and paste actual results (preferably inside code /code tags.

- Show your actual network performance graphs. Rarely maxing could mean too many things, though obviously 20KB/s is problematic.

- Google "ZFS breathing" if you don't already know what it is.

- Look up iperf if you don't already know what it is and post back with results.

 

[cyberjock]

Your hardware looks like you should be able to saturate Gb easily. I'm not sure how powerful the CPU is, I don't keep up with AMD CPUs. But 20kb/sec seems to be a major problem.

Did you by chance enable jumbo packets or are using ZFS compression?

 

[Idiotzoo]

Thanks for the responses, and apologies for not providing more useful information. I'll get on to that.

I haven't got Jumbo frames enabled at the moment, and no ZFS compression. The system can, and sometimes does, get a decent throughput. But it's not consistent, and the very slow transfers occur when moving a lot of small files. My test case for that is a windows profile folder. A few thousand files, most of them tiny.

Interestingly all I find when I google ZFS breathing is people on forums talking about ZFS breathing and suggesting folk google it ;)

 

[cyberjock]

For small files performance will be horrible. This is a limitation of the multiple handshakes and work performed on both sides when creating, writing and closing files on remote storage and there is no way to improve it. If you are getting good speeds copying single large files then I'd say everything is working normally. I'd expect that you can get at least 60MB/sec with those DD values and most likely at least 90MB/sec.

 

[Idiotzoo]

That what I was beginning to suspect. Interestingly I've done the same test on Freenas and Openfiler with the same hardware. Freenas is better, although interestingly Openfiler was faster over iSCSI. Apologies for not going about this properly. I know better than that :(

Here are my DD tests:

dd if=/dev/zero of=testfile bs=1024k count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 169.577820 secs (309172508 bytes/sec)

dd if=testfile of=/dev/null bs=1024k count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 109.365290 secs (479391587 bytes/sec)

iperf is reporting 71MB/s from the client I'm currently on, which matches file transfer speeds. My normal desktop doesn't have a crappy switch in the way and manages over 100MB/s for large files over cifs/SMB.

So it does look like I'm seeing the expected performance from this system.

One interesting thing is I get faster small file copies to our win2k8 VM, hosted on an old dual core AMD CPU using KVM. It has a reasonable LSI card card with raid 5. It's never been fast because of the i/o being cpu bound. Could it be I'm seeing file operations happening to cache, not being written through to disk as ZFS does?