Chelsio T3 10Gbps Ethernet Cards pushing slow read speed

[HenryMNguyen]

Hi guys,

When running CrystalDiskMark, im getting the following results. Im running RAIDz2, could that be causing too much overhead to the CPU and therefore slowing the disk speed?

My specs are as follows:

CPU: Intel Xeon E3-1225 V5 3.3 GHz LGA 1151 80W BX80662E31225V5 Server Processor

MOTHERBOARD: Supermicro MBD-X11SAE-O Server Motherboard Configurator
https://www.newegg.com/Product/BOM.aspx?Item=N82E16813183076

32GB RAM: Kingston ValueRAM 16GB (1 x 16G) DDR4 2133 Server Memory ECC DIMM (288-Pin) RAM KVR21E15D8/16
https://www.newegg.com/Product/Product.aspx?Item=N82E16820242215

STORAGE: 8x8TB WD RED running RAIDz2 where 2 drives are redundant drives.
CACHE: Samsung 840 125GB SSD

NETWORK: Chelsio T3 10Gbps Cards on both ends running Chelsio S320 driver

 

[Chris Moore]

Im running ZFS2

Do you mean RAIDz2?

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

CACHE: Samsung 840 125GB SSD

What made you decide to do this?

What is the purpose of the storage? Why are you trying to get this speed. Configuration depends on purpose.

 

[HenryMNguyen]

Yes, I do mean RAIDz2. This set up is meant to be a media production box. I want to be able to edit my photos and video directly from the box.

 

[Chris Moore]

Yes, I do mean RAIDz2. This set up is meant to be a media production box. I want to be able to edit my photos and video directly from the box.

For normal file read and write, there is no need to have a cache drive. That can actually limit the system performance, especially since it is only a SATA drive.
The other issue is that performance of a single RAIDz2 vdev is roughly equivalent to a single drive in the vdev. Since you only have one vdev in your pool, you are limited to generally the speed of one of the hard drives that make the vdev. The thing you need, if you want to get more speed, is more vdevs. To come close to 10Gb speed, you will likely need 24 drives like the ones you are using now divided into 4 vdevs, configured as 6 drive RAIDz2 vdevs.
Your SATA SSD would limit maximum performance of the pool to 4Gb/s.

Im running RAIDz2, could that be causing too much overhead to the CPU and therefore slowing the disk speed?

Did you look at your CPU utilization while conducting your test?

Here is a peek at some of the stats from my system during a file transfer:

Just for comparison.