Low performance in RAID0 with 10Gb NIC: FreeNAS is a Solution?

[cronos11]

Hello,

I am not sure if I can consult this question here. If it is not possible, I apologize to the moderators. I have a problem with RAID0 and Ethernet transfers in Windows 10 and I am thinking about using FreeNAS to try to solve it.

I have two computers with a RAID0 (striped volume) in each one.

PC-1
HDD: Seagate Exos x10 hard drives in software RAID0. (10TBx4)
NIC: Asus XG-C100C 10Gbps ethernet card.
CPU: i7 4790K with Z97 chipset and 8GB of RAM.
SO: Windows 10 x64 Pro


PC-2
HDD: Seagate Ironwolf hard drives in software RAID0. (8TBx6)
NIC: Asus XG-C100C 10Gbps ethernet card.
CPU: i3 4160 with Z97 chipset and 8GB of RAM.
SO: Windows 10 x64 Pro

I use dynamic disk RAID0 (striped volume) of windows 10 in the 2 computers. With AHCI controller fixed in UEFI.

When I run the cristaldiskmark test I get good values:

PC1

PC2

But when I try to copy a file from PC-1 to PC-2 (or vice-versa), the copy speeds are much lower than the theoretical ones.

* PC-1 to PC-2: copy one file of 35GB I get a 530MB/s
* PC-2 to PC-1: copy one file of 35GB I get a 600MB/s

If I use RAMdisk software I get this speeds:

PC1

PC2

And when I try to copy a file I get the right speed:

* PC-1 to PC-2: copy one file of 5GB I get a 1.000MB/s
* PC-2 to PC-1: copy one file of 5GB I get a 1.000MB/s

Does anyone know why this happens? I do not understand why this happens!
I would like to get 1GB/s copy speeds from my RAID0 volumes between PC-1 and PC-2. Is this possible?

If I use freenas for PC-2, can I solve this problem?
Is there a bottleneck in Win10?
Is there a bottleneck in SATA hardware?
Is it mandatory to use a hardware RAID controller on both computers to avoid this problem?
I can install a RAID controller on PC-1 because I have many PCIe slots. But I can not install a RAID controller on the PC-2 (mini ITX) because I only have one PCIe x16 slot (+ mini-PCIe 2.0 x1)

Thank you very much!

 

[garm]

According to the spec the x10 drives has a max random read/write rate at 170 MBps and according to my calculator 4 x 170 is 680 MBps. So everything seams to be @ spec to me.
I don’t know the innerts of Microsoft storage spaces but the key to increased performance when it comes to spinners is to have as many heads as possible reading as many parts of the problem as possible at the same time. This is achieved in ZFS by having many vdevs in contrast to having few. For the same net storage space that would be many small drives instead of a few larger drives. What I do is a simplified calculation of expected network performance divided by the drives random read/write and that is my vdev count. This isn’t the whole picture, but I don’t care. So for the 170 MBps drives you have this would mean 10 Gb / 170 MB = 7,3 ish eg you need 8 vdevs to saturate a 10 GbE interface. Your striped array is only presenting the equivalent of 4, thus you get half the bandwidth. ZFS and FreeNAS won’t bend the laws of physics. You can however increase the performance of the pool under certain conditions by having large enough ARC and a dedicated ZIL. Block level compression can also have a significant performance boost. Writing half the data takes half the time. ZFS is not built for speed, but reliability. In doing so some interesting performance mechanisms have emerged. Unfortunately the hardware you appear to have dosent suit FreeNAS. You most probably dont have enough RAM for a 40 TB pool, the i7 dosent support ECC RAM and the 10 GbE cards you have lack FreeBSD drivers. Also, the Z97 is proabobly attached to a consumer grade motherboard ill suited for reliable storage, not to mention the fact that your drive configuration lacks the redundancy needed by ZFS to heal broken data.

 

[cronos11]

According to the spec the x10 drives has a max random read/write rate at 170 MBps and according to my calculator 4 x 170 is 680 MBps. So everything seams to be @ spec to me.
I don’t know the innerts of Microsoft storage spaces but the key to increased performance when it comes to spinners is to have as many heads as possible reading as many parts of the problem as possible at the same time. This is achieved in ZFS by having many vdevs in contrast to having few. For the same net storage space that would be many small drives instead of a few larger drives. What I do is a simplified calculation of expected network performance divided by the drives random read/write and that is my vdev count. This isn’t the whole picture, but I don’t care. So for the 170 MBps drives you have this would mean 10 Gb / 170 MB = 7,3 ish eg you need 8 vdevs to saturate a 10 GbE interface. Your striped array is only presenting the equivalent of 4, thus you get half the bandwidth. ZFS and FreeNAS won’t bend the laws of physics. You can however increase the performance of the pool under certain conditions by having large enough ARC and a dedicated ZIL. Block level compression can also have a significant performance boost. Writing half the data takes half the time. ZFS is not built for speed, but reliability. In doing so some interesting performance mechanisms have emerged. Unfortunately the hardware you appear to have dosent suit FreeNAS. You most probably don't have enough RAM for a 40 TB pool, the i7 dosent support ECC RAM and the 10 GbE cards you have lack FreeBSD drivers. Also, the Z97 is proabobly attached to a consumer grade motherboard ill suited for reliable storage, not to mention the fact that your drive configuration lacks the redundancy needed by ZFS to heal broken data.

Thank you very much for your answer. Sorry for my poor english, I'm spanish speaker.

I want to clarify that hard drives reach about 240MB/s and not 170MB/s. You can see the crystalldiskmark test where I get about 950MB/s in RAID0 mode. 950/4 = 240MB/s. I understand therefore that your calculations would be different.

I have done additional tests. I send you captures of the copy graphics so you can analyze their stability. I think that the origin of the problem is likely to be the windows operating system. What is your opinion about this?

Indeed my data is important and I do not mind spending some money. I'm going to buy a Fujitsu CP400I RAID controller (it has no cache memory). I will install the RAID controller on PC-1 and I will repeat the test. The problem I have is that I can not install a RAID controller on the PC-2 because it is a mini computer (mini ITX) and I only have one PCIe available that I need to use for the 10Gb NIC.

with a single RAID controller can I get 1GB/s transfers? or do I need two RAID controllers?

If I have not misunderstood you say that my hardware is not suitable for installing freeNAS and using software RAID0 mode, is it correct? The most important thing is to add more RAM? with 16GB on each computer would be enough?

Thank you very much!

Copy from RAID0-PC2 to RAID0-PC1:

Copy from M2-PC2 to RAID0-PC1:

Copy from Ramdisk-PC1 to RAID0-PC1 (internal copy):

Copy from Ramdisk-PC2 to RAID0-PC1:

Copy from Ramdisk-PC2 to Ramdisk-PC1:

 

[garm]

I’m going on the spec sheet for your drives. They specify max random read/write speed to be 170 MBps. That you have sequential read/writes higher then that in non-real world conditions won’t change anything. https://www.seagate.com/files/www-content/datasheets/pdfs/exos-x-10DS1948-1-1709-GB-en_GB.pdf

 

[cronos11]

I’m going on the spec sheet for your drives. They specify max random read/write speed to be 170 MBps. That you have sequential read/writes higher then that in non-real world conditions won’t change anything. https://www.seagate.com/files/www-content/datasheets/pdfs/exos-x-10DS1948-1-1709-GB-en_GB.pdf

thanks for the clarification. but if I make a direct copy of a 30GB file, is that a sequential copy or a random 4K copy? I thought that this type of copy was a sequential copy and therefore the speed that had to take into account was the 249MB/s instead of 170MB/s

I'm going to use RAID0 exclusively to copy files from PC1 to PC2. I'm not going to set up a server or anything like that. Should I assume that the speed of reading and writing valid for software RAID0 are random and not sequential?

thank you!

 

[Johnnie Black]

* PC-1 to PC-2: copy one file of 35GB I get a 530MB/s
* PC-2 to PC-1: copy one file of 35GB I get a 600MB/s

In my experience you need a little more raw performance than the speed you'll get with a copy, i.e., PC1 can read/write at around 900MB/s, you'll need raw performance of around 1100/1200MB/s to sustain a 1000MB/s copy.

For example, my desktop has dual 970 EVO NVMe devices in RAID1, Windows explorer copy can write at around 600MB/s, though I'm using ReFS with checksums enable and that has a performance penalty, still the devices can handle substantially better sustained writes, for reads I do get around 1100MB/s.

 

[tres_kun]

I will suspect the i3 4160
I have an i3 4150 in my nas
Cached content can be read at 700MB/s
Sending data over ethernet takes more resources than local copy