Poor copy performance between volumes

[downingjosh]

FreeNAS 8.3 p1 x64
Dell T620
1x Xeon E5-2620
32 GB DDR3 1600Mhz
Network - 1x Intel I350 DP
Storage - PERC H310 RAID Controller (no configuration on the controller, passing through to FreeNAS directly)
4x WD Red 3 TB SATA drives in zfs RAID 10 exported via NFS
4x Samsung 840 Pro 256 GB drives in zfs RAID 10 exported via NFS
1 Windows Server 2012 on VMware vSphere 5.0 on a separate Dell T320

We have had some write performance issues with this machine so far. Running rsync from another physical machine on the network to a Server 2012 VM is very slow (copy occurs at around 4.5-6 MBps). The Server 2012 OS sits on the SSD array with a data drive on the WD Red array. Both are NFS exports from FreeNAS to vSphere connected via a HP Procurve gigabit switch.

When running tests on the volumes themselves via

Code:

dd if=/dev/zero of=/mnt/ARRAYNAME/zerofile.000 bs=1m count=40000

I get 821 mbps on the SSD volume and 225 mbps on the WD Red volume. Nothing wrong with that.

When I copy that 40 GB file between the volumes using dd, I get much poorer results. Running

Code:

dd if=/mnt/flash1/zerofile.000 of=/mnt/RedDisks/zerofile.000

I get 42 mbps copying from the SSDs to the WD Reds and 37 mbps copying from the WD Reds to the SSDs.

The RAID controller isn't great by any means, but I can't imagine it would cause that kind of bottleneck. Any ideas?

 

[downingjosh]

No problem, just bad testing methods

After some reading, I realized that not specifying the block size & quantity in the dd command (I didn't realize how much that affected performance, thought it was just for determining the size of the output file created) for the volume-to-volume copy wasn't producing an accurate, comparable test. I reran the tests with the same block size/quantity numbers and got the following results.

Code:

dd if=/mnt/RedDisks1/zerofilenew.000 of=/mnt/flash1/newzerofile.000 bs=1m count=40000

Red Drives to SSDs - 247 MBps

Code:

dd if=/mnt/flash1/newzerofile.000 of=/mnt/RedDisks1/zerofilenew.000 bs=1m count=40000

SSDs to Red Drives - 221 MBps

Those numbers are limited by the read/write speed of the WD Red drives as noted in their tests:

Code:

dd if=/mnt/RedDisks1/zerofilenew.000 of=/dev/null bs=1m count=40000

Red Drive Read test - 297 MBps

Code:

dd if=/dev/zero of=/mnt/RedDisks1/zerofilenew.000 bs=1m count=40000

Red Drive Write test - 225 MBps


So it appears that there isn't actually a problem here. Nothing to see folks, move along...