Upgrade time - new pool layout options

[J8K]

I'm in the planning phase of expanding/increasing my storage. But I'm having issues deciding on which option to go for..

My current setup:
Cpu: i3 6100
MB: Supermicro x11ssh-ln4f
Ram: 64GB ECC
Hdd: 8x3TB WD Reds in RaidZ2
PSU: 1000w EVGA G2
Case: 4u chassi with 15 internal drive bays

My plan is to use 10TB WD Reds and remove all of my 3TB disks. I have already started to purchase new drives in different batches.

Options:
1. Pool with 3-way mirror (for my/family very important files) + pool with 11 disk RaidZ3. (maybe even an extra spare in last bay?)
2. Pool with 3-way mirror + pool with 2 vdevs of 6 disks in each RaidZ2.
3. Pool with 3-way mirror + pool with striped mirrors. (not my top choice, 50% lost space..)

My main concern right now is the time/stress with the re-silver and scrub of option 1 vs 2. I have seen similar builds with both options but none with that big drives that gives data/time to compare.
I'm in favor for option 1, but option 2 should be better in terms of scrubs etc..

Do you guys have any good advise to give?


And yes, I have backup :)

 

[danb35]

My main concern right now is the time/stress with the re-silver and scrub of option 1 vs 2.

Why does this concern you?

 

[J8K]

Why does this concern you?

It might be that I'm a little paranoid about it.. but I have seen too many post about people saying that it will stress the drives for a longer time and that UREs might happen with larger drives (or increased chance at least) and that wide vdevs are bad etc..
But on the other hand.. I have been running zfs for many years now on several systems without any issues. So maybe I shouldn't worry at all.

 

[danb35]

I've seen this position stated repeatedly, but never really supported. As best I can tell, its origin is this blog post, which rather conclusorily asserts, but never demonstrates, this problem. Yes, it's true that resilvers (but not scrubs) will take longer. What of it? Well, this leads into the "RAIDZ1 is dead" zealotry that's been pretty pervasive, with people going so far as to say that "when you resilver your RAIDZ1 pool, it will fail due to a URE on a second disk" (which is simply nonsense). A few points to consider:

• Disks are designed to do i/o--that's pretty much their entire purpose for existing.
• Scrubs are recommended routine maintenance for your pool, so much so that they're scheduled by default in FreeNAS (albeit not frequently enough, IMO)
• A scrub reads all data on the pool, compares it to its checksum, and repairs errors if there's sufficient redundancy to do so.
  • Corollary: a scrub will take about the same amount of time on a mirrored pool as on a parity RAID pool, assuming comparable disk characteristics and the same amount of data. The higher IOPS of the mirrored pool could reduce the time a bit, but scrubs are now heavy on contiguous reads, so this shouldn't result in major differences AFAIK.
• A resilver puts the same stress on the vdev in question as a scrub.
• The length of time for a resilver to complete is proportional to the amount of data stored on the vdev in question.
  • Corollary: A resilver will typically be faster on a mirrored pool.

So, no, I'm not concerned about the "stress" of a resilver causing problems with the remaining disks in the vdev--and I've resilvered my pool many times, for a variety of reasons, without a problem. I don't believe there's any significantly-increased risk of a disk failing at this time compared to at any other time. I'll admit that's just my belief, without any real data to support it--but I haven't seen any real data to contradict it either, and the points noted above would tend to support my belief.

Now, I'm not saying to use RAIDZ1; I agree with the general consensus here that it isn't (generally) a good idea with disks > 1 TB. But, as with non-ECC RAM, the risk is overstated.

 

[Chris Moore]

Pool with 3-way mirror

I would say this is not needed. If you want triple parity / redundancy, just use RAIDz3 all the way across. The purpose of mirrors it to use many mirrors to get high IOPS. For the number of disks you are discussing, just use two seven disk vdevs of RAIDz3 (14 disks and an open bay) so all of your data is protected against three drives of failure and you still have two vdevs of IOPS. You are over-complicating something that should be simple.

 

[danb35]

use two seven disk vdevs of RAIDz3

I'd probably say RAIDZ2 instead, but would otherwise generally agree--though I think there are also mismatched disk sizes to contend with.

 

[Chris Moore]

I'd probably say RAIDZ2 instead

For me, I use RAIDz2, but the OP was talking about a 3-way mirror, so I guess that was the redundancy he wanted.

 

[HoneyBadger]

Well, this leads into the "RAIDZ1 is dead" zealotry that's been pretty pervasive, with people going so far as to say that "when you resilver your RAIDZ1 pool, it will fail due to a URE on a second disk" (which is simply nonsense).

There's one immutable truth about the whole RAIDZ1 thing, and that's this: a URE during a resilver of a degraded vdev causes data loss. This is similar to the concept of "non-ECC is okay" - unless you verify your data on the initial copy to ZFS, you do have a very short window of time where you could see data loss if the data is corrupted in-memory before it is checksummed and made redundant.

Does it mean your whole pool is toast if either of these things happen? Of course not. ZFS is extremely paranoid about storing its metadata, to the point where (I believe) the standard configuration holds no less than six copies of the stuff that causes pool death. But whatever data that gets hit by a URE or bitflip is most likely corrupted and gone. Maybe that means a single image, a portion of a document, or a few frames of a video are garbled. Maybe this means a record in a database is mangled. Maybe it was a block commit to a ZVOL, and it renders an entire VMFS filesystem unmountable. Who knows?

How much this impacts you should be considered strongly by the end-user; and only the end-user can tell us what that data is worth, in terms of lost dollars, unexpected downtime, or emotional impact.

A few frames of a pirated or ripped DVD? No big loss, you'll get it again.
Treasured images of your family photo collection? Potential tragedy.
Corrupted DB for a business? Massive reputation and financial impact.

For most people, FreeNAS is their backup. But you have to remember that RAID is not backup, and neither is ZFS vdev redundancy. Practice defense-in-depth. Add an additional layer of protection. Make backups, and test them. Follow the 3-2-1 rule if you're really concerned:

3 copies of the data
2 different media
1 copy offsite

Statements like "RAIDZ1 is dead" or "you must use ECC memory" and "mirror your SLOG" are meant to be risk reduction. Some people have taken them to very serious extremes; but in my opinion, if we weren't extreme about our data protection, we'd be running mdraid on ext4 or BTRFS or some other solution that plays it more fast-and-loose with the data. Take them with a grain of salt, but be careful not to strawman the argument.

 

[danb35]

For me, I use RAIDz2, but the OP was talking about a 3-way mirror

With either of these, you have two disks' worth of redundacy--two disks fail with no data loss, but you lose data when a third disk fails.

 

[danb35]

There's one immutable truth about the whole RAIDZ1 thing, and that's this: a URE during a resilver of a degraded vdev causes data loss.

...unless it's in metadata. In a traditional RAID5 array, that means instant death of the array (which is the biggest reason usually stated for the claim)--but in RAIDZ1, with at least two copies of all metadata, it may yet be recoverable.

Again, I agree with the standard recommendation, but not with the lengths to which it's been taken by some--those are pretty much FUD.

 

[J8K]

Thank you all for the feedback!
I have realized the I might have over-complicated things a little bit as Chris said.. I'll skip the mirrors and stick to a RaidZx setup.
The 7+7 option sounds like a good option for me. Since I got the free bays to start with 1 vdev and migrate data from the old one before adding the other vdev.

One last thing.. :) Why would you go on a 7 + 7 RaidZ3 instead of the 11 disk RaidZ3? (given that I dont need the iops of 2 vdevs) Both options will give me 8 data drives and I need to buy 3 less if I go with one 11 disk vdev.