Why "Optimal" number of disks?

[rihad]

You can still see a help tip showing this "rule of thumb" recommendation in the current 11.2 FreeBSD installer when choosing an automated raidz setup.

 

[Chris Moore]

You can still see a help tip showing this "rule of thumb" recommendation in the current 11.2 FreeBSD installer when choosing an automated raidz setup.

Guidance like this tends to last for a while because it isn't hurting anyone.
Your assessment that it's wrong is the problem. Why are you having a problem with the guide?

 

[danb35]

You can still see a help tip showing this "rule of thumb" recommendation in the current 11.2 FreeBSD installer when choosing an automated raidz setup.

So why are you coming here to discuss an issue you have with the installer for a different product?

 

[rihad]

Guidance like this tends to last for a while because it isn't hurting anyone.
Your assessment that it's wrong is the problem. Why are you having a problem with the guide?

I was just pointing out a logical fallacy. IIRC it is claimed that raidz1 should be chosen with 5 disks, but raidz2 for 4 disks, to make optimal use of space & speed. Well, 4-disk raidz2 is arguably slower and wastes more space than 4-disk raidz1 would.

So why are you coming here to discuss an issue you have with the installer for a different product?

Just to show how you mistakenly believed that the recommendation was 4 year old and no longer relevant.

 

[danb35]

It is no longer relevant to FreeNAS. If you have an issue with the FreeBSD installer, take it up with them. FreeNAS has not used the recommendation in question since before this thread started, meaning it was out of date even then (and much more so now).

The recommendation was (five years ago or so) that parity RAID vdevs should have 2^n+p members--some power of two, plus the number of parity disks for the RAID level. So RAIDZ1 would have 3, 5, or 9 disks; RAIDZ2 would have 4, 6, or 10; RAIDZ3 would have 5, 7, or 11 (with the latter discouraged as vdevs > 10 disks wide have other performance issues). The reason for this had to do with space utilization, block allocation, and the like, but with the advent of compression that's turned on by default (which, again, was several years ago), that reasoning went out the window. The recommendation had nothing to do with performance as such. So with that background, this:

IIRC it is claimed that raidz1 should be chosen with 5 disks, but raidz2 for 4 disks, to make optimal use of space & speed.

...incorrectly understands both the recommendation and its effects.

Well, 4-disk raidz2 is arguably slower and wastes more space than 4-disk raidz1 would.

Parity is not wasted space, it's used, and for very good reason. Whether it's valuable to you is something you'd need to decide, but it's still being productively used. The issue with non-optimal vdev sizes is that large amounts of space would actually be wasted--they wouldn't be used for data, they wouldn't be used for parity; indeed they wouldn't be usable at all.

 

[Chris Moore]

I was just pointing out a logical fallacy.

Your failure to understand the purpose of redundancy is not a logical fallacy, it is a failure on your part to have done your research.

Read these resources and you might begin to understand:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

If you read them before, read them again, because you must have skipped over the important parts.

 

[rihad]

I should note that good quality SSDs are much less likely to fail than HDDs (does anyone still start using them these days?). Paying a consistent speed penalty preferring raidz2 over raidz1 for 4 SSDs is questionable. Replacing the server with a new one every couple of years is much better than relying on the extra redundancy. And it will probably be more reliable & fast for the same price, as technology is constantly improving. Having off site backups is still a must even with 10-way arrays.

 

[Ericloewe]

I should note that good quality
SSDs are much less likely to fail than HDDs

There's a lot more nuance to the matter than you suggest. Like HDDs aren't magic, neither are SSDs.

Paying a consistent speed penalty preferring raidz2 over raidz1 for 4 SSDs is questionable.

What speed penalty? You say that as if it's a massive difference. The bottleneck is almost always somewhere else.

Replacing the server with a new one every couple of years is much better than relying on the extra redundancy

Now that statement is simply absurd. Those are nearly orthogonal questions.

And it will probably be more reliable & fast for the same price, as technology is constantly improving. Having

In the fantasy world of marketing, perhaps. Reliability hasn't meaningfully changed. Price tends upwards and processing power has been close to stagnated for nearly a decade.

Having off site backups is still a must even with 10-way arrays.

That is true.

 

[rihad]

What speed penalty? You say that as if it's a massive difference. The bottleneck is almost always somewhere else.

See this raidz1 vs raidz3 5-disk comparison:

5x 256GB raid5, raidz1 931 gigabytes ( w= 817MB/s , rw=610MB/s , r=1881MB/s )
5x 256GB raid7, raidz3 464 gigabytes ( w= 424MB/s , rw=316MB/s , r=1209MB/s )

 

[Ericloewe]

There are several problems with that:

• You're quoting numbers for RAIDZ1 and RAIDZ3 instead of RAIDZ2, which was being discussed.
• The benchmark is absurd. By issuing tiny sync writes, it's crashing against the disks' IOPS limitations, not any sort of CPU bottleneck. RAIDZ3 issues twice as many disk write ops as RAIDZ1 for small writes (ZFS writes a minimum of p+1 sectors, where p is the party level). Not-coincidentally, the results show RAIDZ3 with half the performance of RAIDZ1.
• This scenario is fundamentally absurd because you'd be using an SLOG to handle the incoming sync writes.

 

[rihad]

SLOG for pools that are already on SSD? That's a use case I haven't heard of. It's like RAM cache for RAM accesses ) From performance standpoint an intent log might only makes sense if it's on a faster disk (currently slog on SSD vs. pool itself on spinning disks).

 

[danb35]

I should note

...several other things that continue to be completely unrelated to this thread.

 

[Ericloewe]

SLOG for pools that are already on SSD? That's a use case I haven't heard of.

SSDs are not equal. Maybe if you had a pool with P4800X drives the SLOG would be useless, but it definitely isn't if your pool is made up of Samsung 860 Pros.
Also, the numbers you quote were obtained using mechanical hard drives, so your assertion doesn't make sense.

 

[rihad]

Also, the numbers you quote were obtained using mechanical hard drives, so your assertion doesn't make sense.

SSD, right after the spinning disks.

 

[Ericloewe]

SSD, right after the spinning disks.

Oh yeah, I missed that when I was reading it on the phone. Still, my point stands: The workload is IOPS-bound (except for reads, which between ZFS bundling transactions into groups and SSDs being really good at scale approximately linear with the total number of disks in the RAIDZ vdev, which matches the rule of thumb), so it's not a realistic test.
If it had been done with enterprise SSDs tuned for highly-random writes, the test might be meaningful to generic RAIDZ performance, but not with Samsung 840 drives. Anandtech's reviews of SSDs are pretty illustrative of just how horribly consumer SSDs deal with these workloads.

 

[Chris Moore]

So the general recommendation for the number of disks goes like this

This thread started five years ago. Still a very interesting and informative read. I don't know how I got here...