Mirror RAIDZ Volume

[Steve Lawlor]

Forgive me if this is a repeated question. I am new to FreeNAS and have been searching the forums for an answer for a while now. Maybe I am just not understanding the terms or something...

I am building a FreeNAS box with 8 hard drives. I want to set up 2 raidz volumes that each use 4 drives. I then want to mirror the 2 volumes, very similar to how an HP LeftHand SAN works. Is this possible and, is this a good idea if possible? I have read that this might be possible using replication but that seems like quite a bit more overhead than a simple mirror set. Any insight would be greatly appreciated.

 

[Nick2253]

I don't think you mean mirroring; are you talking about striping? Mirroring RAIDZx vdevs doesn't really make much sense. You already provide redundancy by using RAIDZx.

Let's step back a minute: what is your goal here? If you're trying to make a backup of your data on the same server, then I'd use replication to do it. If you're trying to make your array more robust against drive failure, I'd put all eight drive in one RAIDZ2 or Z3 vdev. If you're trying to get more performance out of your array, then I'd stripe the two RAIDZ1 vdevs.

 

[Steve Lawlor]

Thanks for the quick reply Nick2253. My goal is to make the array more robust against drive failure and not really a backup. I am not opposed to using all 8 drives in a z3 config. I was just curious if FreeNAS supported a configuration like that. I called it a mirror because I want an exact copy, in real time, on both volumes. The way I understand striping is data is written across all drives or drive groups. Replication does not sound like it is real-time and sounds like it is a bit more overhead (although I admit I know nothing about how it works in the background). Sounds like maximum drive failure protection is best achieved with z3 then?

 

[Nick2253]

Your understanding of mirroring vs striping is correct. It was just an odd request, and the most likely explanation in my mind was that you were confusing the two. My apologies.

If you want truly maximal redundancy, you could do triple mirrors. But I would say that realistically, RAIDZ3 is the right way to go: you get very good redundancy, and better space utilization over mirrored RAIDZ1. Obviously, your acceptable risk is up to you, so in the interest of full disclosure: mirrored RAIDZ1 vdevs gives you the ability to lose three disks without problem, and if the wrong fourth disk fails, you'll be in trouble. But you could lose up to 5 disks without problem. RAIDZ3 on the other hand allows you to lose 3 disks without problem, but a forth disk lose will always cost you your pool.

Honestly, with only 8 disks, I'd probably recommend RAIDZ2 as a reasonable compromise between redundancy and space: you can lose two disks without issue, after all. However, if uptime (and by extension, redundancy) is extremely important to you, I'd go with RAIDZ3.

As always, RAID is not a backup: it's an uptime protection against disk failure.

 

[Steve Lawlor]

Thanks again. I agree, RAID is not a backup. That is what my second FreeNAS box will be for... If I remember correctly, with z3 I can lose up to 3 drives as long as all 3 are not the parity drives, correct? Same with z2, just two drives...

 

[Nick2253]

The parity is spread across the drives. There is no distinction between "parity drive" and "non-parity drive". In RAIDZ3, you can lose any three disks, and your array will still be operational.

 

[Steve Lawlor]

That settles it then. Sounds like z3 for me at least.

 

[Stux]

mirrored RAIDZ1 vdevs gives you the ability to lose three disks without problem, and if the wrong fourth disk fails

I don't think you can do mirrored raidz1 in ZFS.

Your choices are stripe, mirror or raidz1, raidz2 or raidz3.

In effect a stripe is a 1 disk mirror. You can add another drive to it to convert it to a mirror, and then add another drive to convert that to a 3 way mirror. I think you can probably do 4way, but that is well and truly overkill.

THEN when you have more than one vdev you can ONLY stripe them. Its not really striping... but the writes will be distributed amongst the vdevs in a way that is effectively striping.

So, vdevs contain redundancy, and striping is the way that vdevs are joined for extra capacity.

Now, if you have a collection of vdevs, be they raidz or mirrors, then if ANY of those vdevs fail your pool will fail.

Thus the redundancy is limited by the weakest point.

In a collection of 10 2way mirrors, if any 2 drives in any of the mirrors fail, you fail.

In 10 way raidz2, any two drives can fail, and you should be okay (providing no URE)

in a 10way raidz3, any three drives can fail.

in 3 5 way raidz1, if two drives fail in any of the raidz1 vdevs, you're hosed.

 

[Stux]

The parity is spread across the drives. There is no distinction between "parity drive" and "non-parity drive". In RAIDZ3, you can lose any three disks, and your array will still be operational.

And more importantly, you can lose 2 drives, and still recover from as many UREs as you have, as long as the UREs are not in the same place on the drives they occur in.

The chances of two UREs being located in the same block on two multi-TB drives in the same vdev is ... remote.

 

[Robert Trevellyan]

I don't think you can do mirrored raidz1 in ZFS.

Right, with ZFS, the redundancy is at the vdev level, not across vdevs. Above the vdev level, you have pools, and if a pool contains multiple vdevs, they are striped together.

 

[averyfreeman]

Forgive me if this is a repeated question. I am new to FreeNAS and have been searching the forums for an answer for a while now. Maybe I am just not understanding the terms or something...

I am building a FreeNAS box with 8 hard drives. I want to set up 2 raidz volumes that each use 4 drives. I then want to mirror the 2 volumes, very similar to how an HP LeftHand SAN works. Is this possible and, is this a good idea if possible? I have read that this might be possible using replication but that seems like quite a bit more overhead than a simple mirror set. Any insight would be greatly appreciated.

Sorry to revive the necrothread, but I saw your question and realized I had just read this article where the author tested exactly that configuration (and had excellent results):

https://icesquare.com/wordpress/zfs...aidz-vs-raidz2-vs-raidz3-vs-striped/#raidz1x2

So it definitely can be done, and might even be advantageous (best of both worlds - quick scrubs and resilvering due to mirrored drives and fast IO due to striping)

 

[anmnz]

Nope. That article talks about striping RAIDZ vdevs; the OP is asking whether you can mirror them.

 

[garm]

Ninjad

What you linked to has nothing to do with the question and everything to do with the answer. The answer still stands, you cannot mirror vdevs in ZFS

 

[averyfreeman]

I stand corrected!