Performance Issues

[Hermensch]

I got a freenas Setup with the following hw:
16GB Ram
CPU: Intel G3258
4x2TB Ironwolf
3x1TP old HDDs
10Gbit NIC

When I have a raidz1-pool with the 4 Ironswolfs, my average writespeed on big files (movies 11GB) copied from my Windows-PC from Ram-Disk via SMB is max. ~150-200MB/s.
gstat shows that the drives are max. 60-70% busy.

When i do a raidz1 with the 3 1TB old HDDs and do the same copy-test speeds are about the same. gstat shows 100% load on the disks.

CPU usage on all tests max 75% on avg. 60% so CPU is not bottlenecking.

When i do a Stripe pool with either of the drives i get full bandwidth. eg. 3x1TB drive stripe ~275MB/s.

It seems like i do have a bottleneck but i cant find it, it seems like it is the Software?
I am running 11.2 stable.

Maybe someone can help me here. Cant believe 4 Ironwolfs do have the same performace as 3 old 2,5" HDDs :(

cheers

 

[Chris Moore]

Not enough disks. Each vdev provides roughly the performance of the slowest single disk in the vdev. Add more vdevs to the pool to get more performance. All vdevs in a pool should be equivalent to each other.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

 

[Hermensch]

Thanks for the fast reply!

i used to have striped vdevs with 2x1Tb mirrors + 2x500GB mirror in the same system. Performance was about the same 150Mb/s.
So i should have about 250MB write speed? 3 vdev with 100MB/s writespeed -~50MB/s because of not same sizes.

Also what is freenas doing when nothing is bottlenecking?

I do want to have at least 300 Mb/s write/read speed with 4 Ironwolfs and at least 1 Disk redundancy. What Setup could work? Maybe different OS?
Is the Performance of RAID5 same as Raidz1?
It seems nobody with 4 Disks does have 10Gbit NICs in the forums :/

thanks

 

[Chris Moore]

With only four drives, the drives are the limit of performance, so there's no reason to have 10Gb networking.
The thing that you want cannot be done with so few disks

 

[Hermensch]

I reconfigured my stetup with the 4 disks as a stripe(i dont care about the data, just performance).
Now writing to the pool is blazing fast as i want it at about 400 to 500 MiB/s, but reading is slow like 1 drive about 90 to 150 Mib/s when copying/reading big files to my NVME MP510 via SMB over 10GBIT NIC.
Why is only reading so slow?

In my understanding reading must me equivalent to writing in a stripe for big files.
Can anyone give me a hint why? I cant find enough information with only stripes + zfs ?
Is RAID a better solution when i only care about performance with 4 Disks ans reading big files?
setup:
no atime
sync disabled
lz4 compression
2TiB/~7,8TiB used

netdata shows ~60% disk utilization, 40% cpu spikes to ~90%

 

[jgreco]

Read speeds are dependent on many factors. In general, writes can be faster with ZFS because they are staged to RAM and then written to disk, whereas if you are requesting a read for something ZFS didn't precache, it has to go all the way out to the disk for each block it reads, and it may not decide to speculatively read ahead that far if you have limited RAM.

If your pool is busy and if it has been full-ish in the past, you may also be wrestling with fragmentation. This causes significant amounts of seeking which will slow down reads.

 

[Hermensch]

What you are talking about is latency, the time until the data begins to read. But in my case is is about sequential data which is read on bulk/in sequence from the storage. So i think this is not my issue, except ZFS thinks it is a good idea to spread sequential data to random places on the disk.
My transfer of data was the only task so the pool was not busy doing something else. The fragmentation is 2% according to zpool fragmentation .
Is there any deep explanation of how zfs works on the internets?
Topics like this, but just deeper: link
It seems to me that zfs is only made for +20 disks + zlog+ l2arc and random I/O and of course the features of zfs hence my asking for RAID or something else which is capable of streaming sequential data of of +~300 MiB/s for 4 Disks?

 

[jgreco]

What you are talking about is latency, the time until the data begins to read. But in my case is is about sequential data which is read on bulk/in sequence from the storage. So i think this is not my issue

Fine, I just fell off the turnip truck yesterday.

 

[Hermensch]

Fine, I just fell off the turnip truck yesterday.

Sorry for my bad/offensinve formulation but i dont seem to understand it properly why sequential reads must be cached in RAM and Writes not to perform the same speeds. I do seek better understanding but cant find sources or a good explanation for it.
For me sequential reads and writes should be the in the same performance in a Stripe without any caching.

 

[Hermensch]

Read speeds are dependent on many factors. In general, writes can be faster with ZFS because they are staged to RAM and then written to disk, whereas if you are requesting a read for something ZFS didn't precache, it has to go all the way out to the disk for each block it reads, and it may not decide to speculatively read ahead that far if you have limited RAM.

If your pool is busy and if it has been full-ish in the past, you may also be wrestling with fragmentation. This causes significant amounts of seeking which will slow down reads.

i think a failed to mention that the data i am reading is about 40GiB and my RAM only 16GB so caching of the whole thing in ARC is not the cause of the read/write difference.

 

[jgreco]

It's not a complicated idea. Boils down to "RAM is fast."

In a normal NAS, you receive some data via TCP over CIFS, the Samba engine turns it over to FFS or EXT3 for processing, the filesystem talks to the disk drive, writes your stuff, and returns success or failure up the stack. This is why these things are all sucktacularly slow.

With ZFS, when you write, you are actually writing to RAM at write-to-RAM speeds until you fill up a "transaction group" at which point ZFS turns it over to the pool and ZFS starts committing to the pool. In the meantime, your writes are continuing to flow into the next "transaction group." You can have one transaction group being written to and one transaction group flushing to disk. In the right circumstances, this means writes are insanely fast. Your writes generally do not encounter the latency (hey look that word again) of having to traverse the entire filesystem-controller-and-hdd stack.

We talk about latency in the SLOG sticky. It's a different context and a different problem, but it's always an amazing thing to me that storage works anywhere near as fast as it does.

But here's the interesting thing. When doing reads, for any cache miss, meaning something not in ARC or L2ARC, you actually need to traverse a similar set of steps to the big long list in the SLOG sticky. In particular, rather than simply being served from RAM, if you ask for block #12354 of "mybigfile", your hard drives probably need to seek to retrieve that. Now hopefully if you are doing a sequential read, ZFS may predict that you're going to want #12355-#12400 and readahead those into cache. But that will only go so far. Sooner or later, you will hit a point where the file bits you want are not in cache, and everything has to pause for some milliseconds while the drives go fetch the uncached data.