Expandability and loss of redundancy

[andrix]

I'm considering making a NAS with freenas, Got a few questions:

This is my plan, to create a series of Virtual Devices each with 3 drives in RaidZ1 (I know RaidZ2 is better, but for now ignore this). This would give me expandability as I can add a set of 3 drives as a new Vdev and expand the Zpool. However as the Zpool grows, this provides no more redundancy than the 1 disk of redundancy per Vdev; So even if I have say 3 sets of Vdevs and therefore 3 drives of parity, i am still limited to one drive of parity in some cases before the whole Zpool fails.

Is there no way to create multiple zpools each consisting of only one Vdev (eg 3 drives in RaidZ1) and then make them appear as a single [virtual] drive. This would give the benefit of if a single Vdev fails (eg more than one drive fails) Only the data in THAT Vdev is lost, NOT the whole Zpool.

I guess what I am asking is if I can create a SPAN of Zpools either from freenas or in windows, so that I can have multiple Zpools, which appear to me as a single drive, but each zpool is redundant, but If I did lose one Vdev, only the data in that Zpool would be lost.

Heres a link with me trying to illustrate what I'm talking about, and show expandability.

http://i.imgur.com/K2a6u.png

It seems to me that the redundancy in ZFS is only based on low numbers of Vdevs with high number of drives in them; You lose relative safety if you have higher numbers of vdevs with low number of drives in a zpool. Eg, even with mirroring (the best form of redundancy in a Vdev) if you had a zpool of 5 vdevs each with 2 drives mirrored (total 10 drives) you still only have a possible redundancy of 2 drives before the whole Zpool can fail!

I realise that i could use RaidZ2, but if I wanted to keep the initial size of the Vdevs down, but expand later, this would mean say 4 drives per Vdev in RaidZ2, expand up to 3 Vdevs (12 disks) and with 6! disks of parity, I still only get a possible 2 disks of redundancy, Even if I was to mirror I have the same problem, (eg a vdev of 2 disks in mirror)
What I want is to keep the size of the Vdisks down, so that I dont have to buy say 5 x 3/4 TBers upfront, but still have a high redundancy while appearing as a single disk to me.

 

[b1ghen]

There is always 3 way mirrors if you want to be extra safe, not very good at all on useful space though...

 

[andrix]

Yeah, that's the problem though, it still seems that the Zpool idea is unsafe, to me what is needed is redundancy over the VDEV's themselves, since losing a single Vdev kills the whole Zpool, to me it doesn't solve the problem with Raid, just puts it up another level, so instead of disks you have Vdevs as the weak-link.

I saw somewhere else on the forum someone recommending to build a Freenas box using mirroring with Vdevs consisting of 2 disks. Now if you add Vdisks to the Zpool until you have 10 disks, that only gives you a redundancy of at worst 2 disks, does this seem bad to anyone else?

The adding of Vdevs to a Zpool creates a false sense of security, since even if you have tonnes of parity/redundant disks, you are still limited by each individual Vdev having the capacity to bring the whole Zpool down. I can't find much discussion on this subject, is there something I'm missing?

How would you possibly have something with 50+ disks, even with the most secure method of RAID (say 4way mirroring) that's at worst only 4 disk redundancy over 50 disks, even though the vast majority would be dedicated to mirrors.

I think that the Vdevs should not be striped, but rather filled sequentially/separately, with no reliance between the Vdevs, But still have them appear as a single disk. This way you still have some measure of redundancy within each Vdev, but if you were to lose a Vdev you would only lose the data on that Vdev, NOT the whole Zpool, Does no one else share my concerns?

Edit: Just like to clarify that I am a total noob with almost zero linux experience, especially CLI; so I don't want to come across as knowledgeable or anything, I've just been researching and have concerns before moving my data onto FreeNAS/ZFS.

 

[peterh]

if you don't have backups stored somewhere else you live dangerously. No level of raid will guarrantee that you data survives.

 

[StephenFry]

How would you possibly have something with 50+ disks, even with the most secure method of RAID (say 4way mirroring) that's at worst only 4 disk redundancy over 50 disks, even though the vast majority would be dedicated to mirrors.

I might be misreading this, but if you have a 4-way mirror of, say, 16 drives, you'd have 64 drives - of which 3 mirrors can be offline at any point, PLUS one or more drives of the last remaining mirror, depending on the raid level. Say RAIDZ3, so 3 drives could be lost for a grand total of 51 concurrent dead drives out of 64. Sounds pretty safe to me, if you forget about other ways data might be lost. (knock on wood)

But yeah, let's reiterate: RAID IS NOT A BACKUP SYSTEM! It is a high(er) availability system.

I have to admit to trusting some of my RAID systems a bit too much and not bothering with a backup. But those never contain data that is crucial to my or my job's well-being! Important data (and you probably have less of that than you think) should always be backed up and stored in a separate location.

 

[andrix]

Yeah I know about offsite backup, and most of my data is probably not that mission critical. If I have a house fire, I'll probably say fuck it and start again. The important stuff is backed up in multiple locations. That said, I'd like to do anything to not lose ALL of it in one go, a loss of a single HDD worth is much more preferable (over total loss)

What I am talking about, is that there are different ways of looking at this, the 50 drives thing was just a hyperbole, I'll do a lower scale example that's more accurate. Say a Zpool of 7 Vdevs, each with RaidZ1 of 3 disks = 21 disks. Now there is a continuum of looking at this. At one end, you can have one disk fail in EACH Vdev and your data is still safe, any disks after that = total loss of data. The other end, is one disk failure in a single Vdev, data is still safe, If ANOTHER disk in the SAME Vdev fails = total loss of data.

So in a worst case scenario, out of your array of 21 disks, even with 7 disks worth of parity, two disk failures can lead to total data loss. That is the loss of redundancy I am talking about.

The Vdevs THEMSELVES have no redundancy, the Vdevs are effectively put into RAID0 with no redundancy. To Solve this, what should [at least be possible] is the ability to have say RAIDZ1 of the VDEVS, which are themselves made up of individual disks in RaidZ1.

This way loss of a Vdev would not equal total loss of data.

The solution isn't really solved by say using RAIDZ2, as it only adds a single disk of redundancy in a worst case scenario(Which to use my 21 disk array example) is still pretty negligible, even though you would have double the number of parity disks

 

[StephenFry]

the ability to have say RAIDZ1 of the VDEVS

I agree, this would be a great feature to have.

Some time ago, I asked a site-hoster what he uses, and he swears by 10-drive RAIDZ2 pools, which are locally mirrored as well as off-site.
If need be, he adds these pool+mirrors units. This is of course a different kind of expanding than you want to do.

Even though I'm a relative newbie, I'm pretty sure all ZFS users wanting to expand a pool face your exact problem. I hope someone proves me wrong!