TrueNAS RAIDZ[1-2] poor bad speed performance for pool of [3-5]xSSD disks.

[anowosad]

TrueNAS 3xSSD drives pool configured as RAIDZ no compression DEDUP=YES I'm getting above performance. It is constantly dropping with the speed after running about 10s with full speed.
When the same 3xSSD disk are configured as stripe (aka one disk), exactly the same disks the performance is as expected 1Gib (my network).
So, obviously, something is not right with the RAIDZ pool setup.
What am I doing wrong?

CPU is ok. I have 32GB of memory. The only difference on same hardware is stripe/raidz both dedup.

performance drop is >15s and full-throttle lasts about 10s

Version:
TrueNAS-12.0-U7

Latency and disks busy for same RAIDZ Pool with 3xSSD disks.

 

[jgreco]

Welcome to the forums.

In order to help you, please help us. There are guidelines in the Forum Rules, conveniently posted at the top of every page in red, about how to create a good problem report that is more likely to result in high quality useful answers.

For example, please describe your hardware platform, what your use case is, and fill in more detail about what you've done.

Some obvious low-hanging fruit commentary here (but please still answer the stuff above):

Dedup is generally only useful when using ZFS as a storage backend for block storage. We typically recommend a minimum of 64GB of RAM if you are planning to do block storage, but if using dedup, that can be much higher. Dedup memory requirements are generally very much higher than you would expect them to be, and a 32GB system is unlikely to work out. Explaining what you hope dedup to do for you would be helpful.

Turning off compression is generally a bad idea.

The weird throughput artifact is a little interesting, my best guess is that there's some interplay going on between dedup and the write throttle. ZFS uses your system memory as write cache, so can accept writes at blistering speeds, but then when trying to dedup the data, I'm guessing the performance is very poor.

 

[anowosad]

Details have been added to my signature.
Something is wrong with this RAIDZ. I cannot Saturate 1Gbs Network card when the Pool 3xSSD is RAIDZ and compression/no-compression and no dedup. However when the Pool has striped all works as expected. It starts nicely, then after 10s is the speed drop then coming back to full speed and then after a couple of seconds dropping to about 500 Mbs. Statistics show unusual and uneven pool disks utilization with huge (for SSD disks) latensies. Disk % utilization is coming to 100% for 2xSSD but the third is almost not utilized.

 

[anowosad]

The only way so far to almost saturate my 1Gib Network is to create a striped pool from my 3xSSD disks.
Oh well. Looks like there is no silver bullet solution. The performance or endurance/reliability, but you cannot have both.
On the other hand, I'm really disappointed with RAIDZ (50%), RAIDZ2(60%), and Mirror (30%) throughput performance. Only Stripe is giving >90% throughput performance; see picture below. Sample data was 110GB with about 7850 files.

 

[Morris]

You have 3 different types of SSDs and this imbalance can cause very strange issues.

Turn off the dedupe and see how it works in RAID Z

Try a different NIC, some have had issues with Realtek

I can fill take my 10-Gbe LAN to 9-Gb with two sets of 3 WD Helium filled 10-TB drives in RAID-Z and then striped. There is something serious wrong with your setup. Fix the non standard things and it should work better

Good luck,

Morris

 

[jgreco]

10-Gbe LAN to 9-Gb with two sets of 3 WD Helium filled 10-TB drives in RAID-Z and then striped.

Well s*** that's impressive.

 

[anowosad]

You have 3 different types of SSDs and this imbalance can cause very strange issues.

Turn off the dedupe and see how it works in RAID Z

Try a different NIC, some have had issues with Realtek

I can fill take my 10-Gbe LAN to 9-Gb with two sets of 3 WD Helium filled 10-TB drives in RAID-Z and then striped. There is something serious wrong with your setup. Fix the non standard things and it should work better

Good luck,

Morris

"You have 3 different types of SSDs and this imbalance can cause very strange issues."
Striped they work perfectly and saturate my card to almost 1Gib.
"Try a different NIC, some have had issues with Realtek"
The card works perfectly. No need to change it, see above.

"There is something seriously wrong with your setup."
That is the point of that thread. What is wrong with my setup?
"Fix the non-standard things and it should work better"
... "nice" advice. What and how to fix it?

 

[Morris]

A single 7200 RPM spinner can saturate a 1-Gb link. So will a mirrored set of 7200 RPM spinners.

Is a resource running out? I think so.

Are there other issues? I think so.

Is your NIC OK, possibly. It worked OK in one test case.

Are the different disk types an issue? This is outside of best practice.

Why would mirrored disks cause problems? You need to answer that as they should not.

 

[anowosad]

Does anybody here have any experience with TrueNAS fine-tuning? The performance now with 5xSSD disks I have is similar to 3xSDD. Maximum 10 second 1Gib (1Gib is my network speed; the speed drop is on the disk poolside with 3 or 5 SSD disks) transfer and then about 3-5 seconds delay.

 

[anowosad]

I solved this problem on my own.
All works as expected right now.
Yes Realtek NIC is OK and reliable,...

 

[QonoS]

I solved this problem on my own.
All works as expected right now.

Enlighten us please. How did you solve this ?

 

[Christian K]

Yes, @anowosad please enlighten us.
Using a 10GBit Ethernet connection and a single SSD drive (corsair mp510) I noticed strange read performance drops after continuously reading about 4GB of data.