[RESOLVED - by not using 3ware] Slow ZFS performance on 3ware 9650SE Controller

[jgreco]

Try using "bs=262144" instead of "ibs=1024".

I'm thinking it's probably going to use obs=ibs but in any case there's no good reason to "be specific" and the small size isn't helping you.

 

[jgreco]

Actually I'm full of it, it's defaulting to obs=512 and that's just going to chew CPU.

 

[Aaron Ryan]

Try using "bs=262144" instead of "ibs=1024".

I'm thinking it's probably going to use obs=ibs but in any case there's no good reason to "be specific" and the small size isn't helping you.

OK, that made a huge difference, I didnt realize that using a large block size would make such a huge difference.

# dd if=/dev/zero of=/mnt/disk1/nocompress/t.mkv bs=262144 count=32424
32424+0 records in
32424+0 records out
8499757056 bytes transferred in 23.044006 secs (368848933 bytes/sec)
​

Wow, 351MB/s (2.8Gbps) very nice!

 

[jgreco]

It reduces all the overhead and latency involved in repeatedly crossing the kernel/userland boundaries, etc.

 

[Aaron Ryan]

Just to be fair to the 3ware 9650Se-16ML, I put it back in the case with the new Motherboard and Xeon E3-1230v2 CPU
and ran the improved dd command locally:

# dd if=/dev/zero of=/mnt/disk1/nocompress/t.mkv bs=262144 count=32424
32424+0 records in
32424+0 records out
8499757056 bytes transferred in 44.164414 secs (192457146 bytes/sec)
​

So it is a bit faster in the new motherboard and CPU, it got 183MBps (1.4Gbps)

But moving to the SAS2008 based controller doubles the performance to 351MB/s (2.8Gbps)

 

[jgreco]

This is not unexpected. The 3Ware cards are highly optimized towards behaving as RAID controllers, which introduces overhead.