Multiple zpools vs multiple vdevs

[D G]

As a preface, I just finished re-reading Cyberjock's Noob doc on FreeNAS to be sure this wasn't covered in there...I've also searched the forums for a decent discussion on this question, and can't seem to find something truly addressing it. I'm curious if the knowledge exists within the community...I would assume it does.

My question is what are the pros/cons of having 1 zpool with 2 vdevs vs having 2 zpools with 1 vdev each?

I understand well the truth that for 1 zpool with 2 vdevs, if either vdev fails, then the entire zpool is lost. Why is this the most talked about scenario, then, if you can avoid this by having separate zpools? What are the disadvantages of having multiple zpools instead of multiple vdevs? Conversely, what are the advantages of having multiple vdevs in a single zpool?

Searching, I can't find reasons not to have multiple zpools...I have found that it is entirely possible with ZFS and FreeNAS to do this, but doesn't seem to be a common topic-usually we talk about adding vdevs to an existing zpool, instead of making a second zpool. Of course, if there are threads on this very topic that I have overlooked, please do call me out (and kindly provide the links :) ).

Are there performance concerns? Manageability concerns? Underlying, sneaky configuration concerns that I'm unaware of?

 

[pirateghost]

Because people don't want multiple pools generally. They want a pool of all their storage capacity.

 

[D G]

But why not have multiple pools? It doesn't seem like having multiple pools loses you any capacity over having multiple vdevs...? Either way they will have their own parity. When you have multiple vdevs, is that transparent to you when managing the storage in the pool? (I only have one vdevs in a single zpool at the moment-I'm about to add 6 more drives to my system, hence the question)

 

[pirateghost]

Your redundancy lies within your vdevs. Multiple vdevs within a pool shows as just a single pool of vdevs. You don't manage different vdevs, you manage the pool. If you use separate pools, you are limited in space that you can configure for any one service. I prefer to have ALL my space available for any service that I like. I don't want to have to decide what service or data goes to what pool.

 

[D G]

Ah, I could see the argument for that. FYI I currently have 6x3TB in RAIDz2, planning on adding another RAIDz2 vdev of the same drive composition (with 16 GB more RAM of course).

So when you have multiple vdevs in the same zpool, datasets can span across vdevs (which of course wouldn't be possible with separate pools)?

 

[pirateghost]

Exactly

 

[D G]

Ah ok. Are there any other concerns/considerations with having multiple zpools?

 

[pirateghost]

Not that I'm aware of. I don't see much of a benefit for the standard setup

 

[danb35]

ZFS is designed/intended to pool all your storage together, so you can then allocate it, more or less at will, as needed. It's far more convenient to have all of your storage capacity together in one place, where it can be used as needed by whatever needs it, than to have multiple volumes with fragmented free space. If you have a need to limit certain filesystems to a given amount of space (as you would with a partition), you can easily do that by setting a quota for the dataset. But if you need more space in the future, expanding a partition is generally a fairly involved operation. With a dataset, just change the quota and you're done.

The potential downside is that a failure of any vdev within the pool kills the pool, while if you had multiple pools, it would only kill the pool with the failed vdev. If you configure your vdevs properly, you shouldn't have a problem with this (and you should always have backups of important data anyway).

 

[Jailer]

Also Freenas with it's reporting and testing capabilities properly set up lessens the chance of data loss due to pool failure. Things can go sideways all at once but generally there are some early warning signs and FreeNAS will let you know about them so you can have a chance to correct them before it's too late.

 

[Robert Trevellyan]

Are there any other concerns/considerations with having multiple zpools?

with it's reporting and testing capabilities properly set up

which you have to do for each pool if you have multiple pools. The only real benefit I see is the one you mentioned earlier, which is that each pool can survive the failure of the other(s).

 

[danb35]

The only thing that's set up on a per-pool basis is scrubs. Everything else is system-wide, or per-disk.

 

[Robert Trevellyan]

The only thing that's set up on a per-pool basis is scrubs.

Scrubs and snapshots, but fair point, most of the maintenance setup is not per pool.

 

[D G]

Thank you for all of the insight. Still don't know which I will do...I like the idea of having separate pools just so that if something happens to one, I don't lose everything, just half :P

 

[anodos]

Thank you for all of the insight. Still don't know which I will do...I like the idea of having separate pools just so that if something happens to one, I don't lose everything, just half :p

Better yet - make two pools, put them in different servers and use ZFS replication. Then you don't lose anything. :p

My hunch is that an event that takes out a raidz2 vdev on a properly designed server is likely to nuke all the things. You're better off planning an appropriate level of redundancy and backups.

 

[cyberjock]

My hunch is that an event that takes out a raidz2 vdev on a properly designed server is likely to nuke all the things. You're better off planning an appropriate level of redundancy and backups.

+1

If RAIDZ2 can't save you from some kind of totally unavoidable corruption due to a hardware or software issue, having more zpools won't save you anything. It just means that more zpools will be corrupted and unusable. That's all.

 

[brando56894]

I've always had two to three pools setup in my NAS

Previous Setup
Pool for temp storage/seeding torrents (2x1 TB striped)
Single drive for Jails (should be more I know, don't have the money)
Main storage pool (4x3 TB, RAIDZ)

Current Setup
Pool containing one single drive for my jails
Pool for all my storage (4 mirrored sets)

From my understanding, it benefits you to have more vdevs per pool since the vdevs are striped together, but also this increases the risk of failure since one failed vdev can bring down the whole pool.

 

[Montel Bahn]

@ Robert... my understanding is, I'd say "Snapshots are on a 'per dataset' basis", no?

Almost always more flexible to have all the storage in one zpool, nonetheless, realistic use cases for multiple pools might be:

1. Someone has 2 or 3 fast, say 10k, 500GB drives, and purchases a couple of 6TB 5400RPM drives. One might consider a pool of composed of 2 mirrored vdevs, each created by mirroring 1 entire fast 500GB drive with a partition of the same size from the new, slow but huge drives. Thus obtaining a fast raid10 with the 6TB NAS drives basically short-stroked. The rest of the space on the 6TB could be mirrored for cold storage, as a separate zpool. If using 3 smaller drives, one could make a raidz1 pool in a similar manner.

There's no direct performance hit for using a partition rather than a whole drive. But obviously, using 2 partitions in 2 separate zpools could more than half overall performance if both pools are used at the same time(copying from one to the other, or jails contending at the same time, etc).

2. Alternately, with the same example drives from above, simply mirrored 500GB drives, with a copy of most valuable data, perhaps exporting the zpool, or physically disconnecting the cables even, for cold storage. Along with a zpool of the newer 6TB drives mirrored or whatever, for online storage. Obviously offsite backup is far superior(fire, theft, tornado, etc). But it's not irrational to simply copy the data between zpools on internal drives, in the interim. One could even store a zpool inside a waterproof firesafe, maybe down in the family tornado shelter. haha.

3. Personally, I'm thinking of cramming 6 320GB drives and 2 512GB SSDs all into one workstation. I know, I know, but I have the hardware already so I might as well use it right...raidz2 and a mirror I suppose.

I humbly respect pirateghost's direct response; that there are no concerns/considerations to multiple zpools, but it seems understandable why folks have reservations. It is rarely discussed, unlike the importance of sufficient ECC RAM, so I'll selfishly rephrase OPs question because I have no clue myself:

Does anyone know what extra overhead is involved while 2(or more) pools are mounted?
Generally speaking (ignoring partitioning and other exotica)... Twice the memory, for the same performance with 2 zpools?

 

[Robert Trevellyan]

my understanding is, I'd say "Snapshots are on a 'per dataset' basis", no?

True, I was oversimplifying. My snapshots are recursive on the top level dataset, and datasets don't span pools, so I think in terms of a 1:1 correspondence between snapshots and pools.

 

[enemy85]

As far as I can remember, I used to know that multiple vdevs improved zpool performances, while multiple zpools has the same performance of a single one. Is that correct @cyberjock ?