hot spare vdev

[dean.viens]

Hi all,
i read that if you lose a special vdev you lose the whole pool. does this apply to hot spare? it doesn't seem so but I have one mirror that I want to add a single drive hot spare to so if one fails it just replaces it wth the hot spare. i am assuming by the name this is how it works.

 

[Whattteva]

What is this "special vdev" you speak of? I didn't know a vdev can be "special".
A hot spare is just an extra drive that's connected but not really used by the system until a drive fails and it will automatically resilver without any manual intervention.

 

[Arwen]

No, if you loose the hot spare before it is in use, nothing happens to the data in the pool.

If the hot spare is in use, AND it restores redundancy, then fails, your data in the pool is still fine. (Because it restored your redundancy...)


Special vDevs are a different case. Sometimes they are called De-Dup or Metadata vDevs. However, the original design allowed multiple functions to use the Special vDev(s). For example, using a Special vDev for Metadata and small files.

Special vDevs should have as much redundancy as the data pool. For example, a RAID-Z2 pool can loose 2 disks without loosing any data. So a Special vDev should have the same. In general, Mirrors are used for Special vDevs both for speed and reduced stripe size, (because Special vDevs tend to hold smaller blocks). I don't know if Special vDevs support RAID-Zx.

Here are suggested amount of mirrors per data pool redundancy:
2 way Mirror data pool = 2 way Mirror for Special vDev
RAID-Z1 data pool = 2 way Mirror for Special vDev
RAID-Z2 data pool = 3 way Mirror for Special vDev
RAID-Z3 data pool = 4 way Mirror for Special vDev

Complete loss of a Special vDev is total loss of entire pool. Thus, the recommended redundancy as above.

Here is what the ZFS zpoolconcepts manual page has to say on the subject of Special vDevs;
special A device dedicated solely for allocating various kinds of internal metadata, and optionally small file blocks. The redundancy of this device should match the redundancy of the other normal devices in the pool. If more than one special device is specified, then allocations are load-balanced be- tween those devices. For more information on special allocations, see the Special Allocation Class section.

 

[Ericloewe]

What is this "special vdev" you speak of? I didn't know a vdev can be "special".

"Metadata and small block" vdev doesn't roll of the tongue very easily, but would be a more informative name.

 

[Whattteva]

"Metadata and small block" vdev doesn't roll of the tongue very easily, but would be a more informative name.

Huh, strange I've never seen that before even though I've been using ZFS for over a decade. I guess I just never looked hard enough.

 

[Ericloewe]

It's a fairly recent feature, all things considered.

 

[Perry The Cynic]

Speaking of spares and special vdevs, I'm testing around right now and noticing that in a pool with a hot spare and a special vdev, loss (well, hot plug removal) of a disk from the special vdev does not seem to use an available spare to restore redundancy to the special vdev. Is that a bug or a feature? If it's a feature, is there a "special spare" feature I can't find?

Obviously, it's not optimal to take a rotating 10T spare and use it in a 4T SSD vdev, but it beats losing redundancy, doesn't it? And yes, I can manually remove the spare and attach it to the special vdev, but the point of spares is that they get automagically applied to whichever vdev has the failing disk, no?

Cheers
-- perry

 

[sretalla]

noticing that in a pool with a hot spare and a special vdev, loss (well, hot plug removal) of a disk from the special vdev does not seem to use an available spare to restore redundancy to the special vdev. Is that a bug or a feature? If it's a feature, is there a "special spare" feature I can't find?

Just think about that for a minute...

A spinning pool of 10TB HDDs with 2 10TB spares...

Also, a special VDEV of 2 SSDs, let's say 1TB each.

Special VDEV disk 1 fails and is automatically replaced by spinning 10TB HDD... not great, but redundancy and what not.

Then special VDEV disk 2 fails... and is automatically replaced by a 10TB spare... uh oh... now your special VDEV is 10TB and can't be put back to 1TB without a pool rebuild.

Is that what you want/expect?

 

[Perry The Cynic]

Zfs clearly knows that a disk in a vdev is a spare (zpool status says so). I want and expect zfs to not do the auto-expand thing with spares. So in your example, I'd expect the special vdev to run with two 10T spares as a 1T vdev until the failed disks have been replaced.
Of course, clearly my mental model isn't reality - hence my question. How is this scenario supposed to go down?

Cheers
-- perry

 

[sretalla]

How is this scenario supposed to go down?

HDD spares only apply to the data VDEVs as far as I can tell.

I'm not sure that special VDEVs were even considered (since they didn't exist) when the spares code was implemented.

I don't see any reference to spares in any of the documentation of the special VDEVs either.

If you want that to be changed, you'll need to approach the OpenZFS project Github. (maybe adding to this?: https://github.com/openzfs/zfs/issues/12332 ... as it seems nobody cares about it for over a year)

 

[Perry The Cynic]

Thanks for that link. I'll take it up over there when I find time. Meanwhile, I can handle this case manually with enough alerting.

Cheers
-- perry