What are Spare drives in pools useful for?

[JaimieV]

After an experimental phase tuning my then-new Sisyphus NAS build (for 10gigE speed and capacity), I ended up with 3 vdevs of 3x4TB RAIDZ1 for my pool, plus I had a couple of spare slots and drives so popped another two drives in and set them as Spares to the pool.

Whenever ZFS decides to kick out a drive, one of the Spares hops in and takes care of things until I replace the ejected drive. While this is happening the pool is Degraded, not Resilvering. The Spare isn't being used to take the place of the ejected drive like a Replace would. When I Replace the ejected drive with a new one the pool Resilvers as expected.

I originally added the spares because I had literal spare drives and expected them to swap in directly on a fail. That's not what happens.

What's the Spare doing, exactly?
Is there any point in me having them?
Are they actually protecting against a type of failure that could follow after the original drive ejection? I'm certain that if another disk in a VDEV failed I'd lose the pool, so it's not protective in the usual RAIDZ redundancy sense.

Have I just missed this in the FreeNAS docs somewhere, because I can't find much chat about ZFS Spares on the web at all.

 

[sretalla]

What's the Spare doing, exactly?

Sitting there burning up electricity and spinning hours waiting for the chance to jump in... (generally quite wasteful)

Is there any point in me having them?

Since this only kicks in when the "failed" drive actually becomes unavailable to the system... not at any point during the time when the drive begins to show signs of failure in SMART tests, a well managed system will never pull in the spare, because the vigilant admin will have replaced that disk while it was still in a working (but showing SMART errors) state. In that case, a cold spare is possibly a much better option as it's now a fresh disk with no running hours on it.

Where the spare can shine is where you can't easily access the NAS to replace a failing disk in a reasonably short amount of time. This allows you to either react to the SMART reports and manually pull in the spare or it will catch it for you when a disk does actually fail before you were able to intervene.

It's all about your level of comfort with redundancy and time to reach the box.

 

[JaimieV]

I've got a thread somewhere asking why my system appears to kick out perfectly healthy drives, without any SMART errors or warnings that get surfaced to the admin/alerting. Usually a bunch of CAM errors are noted in the BSD logs but FreeNAS doesn't monitor for that. It won't take them back immediately but they test clean and once they've been blanked they'll go back in and run fine again.

So I don't generally get warning - if I did, it's only one set of stairs away in the cellar.

The Spare is doing *something* at those times - a zpool list or the GUI shows it partnered up with the ejected disk. Any idea what? Or is there a way to make it useful by automatically doing a Replace?

 

[sretalla]

Any idea what? Or is there a way to make it useful by automatically doing a Replace?

When you see the spare in play, it will be in the middle of resilvering until that's done, then it's effectively standing in for the "missing" drive, but it's not the same as a "replace", since there are choices to be made on how that should go, so you need to intervene to make that replacement permanent.

If that drive is flapping for some reason, the resilver may be interrupted when the drive becomes visible again (and needs a resilver of its own).

 

[JaimieV]

So it's actually resilvering and I just need to let it go to completion? Nice!

A normal resilver process takes about 8-12 hours after the new drive is added. It's possibly my fallible memory, but I've had to leave the Spare running for a weekend while away and it's definitely still paired up with the ejected drive when I come back - and I'm reasonably sure that the pool is still showing Degraded at that point, well after a Resilver would have completed. I may be wrong! I'll definitely leave it at least a day and check next time.

So would I then choose the missing drive in the GUI and choose to Replace it with the Spare to complete the transition?

(Kindly the disks don't flap, ZFS kicks them out and they stay out)
(Apologies for the capitalisation throughout, I'm trying to make FreeNAS objects/actions obvious since they're all normal English words)

 

[sretalla]

So would I then choose the missing drive in the GUI and choose to Replace it with the Spare to complete the transition?

To be honest, I don't think the GUI has taken spares into account in terms of available functions to manage them, so you're going to be at the command line for all of those operations... back to the article I quoted to make the substitutions.

 

[JaimieV]

To be honest, I don't think the GUI has taken spares into account in terms of available functions to manage them

Heh. Ok! Thanks. I'll run the experiment next time I have a disk ejection, see if my memory is bad or whether the UI is just confused because it doesn't understand the states involving Spares. In which case I'll also log a bug/enhancement request.

 

[JaimieV]

back to the article I quoted to make the substitutions.

I was puzzling over this a little and then found it - in another thread, https://www.ixsystems.com/community/threads/failed-drive-within-zpool.82766/

 

[sretalla]

Oops... Both of your Avatar images are blue(ish) and the topic is similar, so I had confused the two threads... sorry for that.

I mean...

Well done for passing my very cleverly set test to get to the right information. ;)

 

[JaimieV]

Ok! So I've been doing some restructuring, and on my test system had a drive get tagged "faulted" for too many failures. The spare jumped in and resilvered.

Resilver completed, Spare in place as a temp, state still degraded:

Code:

  pool: DataPool0
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: resilvered 1.34T in 0 days 05:53:12 with 0 errors on Sat Mar 21 02:42:32 2020
config:

    NAME                                              STATE     READ WRITE CKSUM
    DataPool0                                         DEGRADED     0     0     0
      raidz1-0                                        ONLINE       0     0     0
        gptid/7abf8085-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8d7ccf17-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/94977190-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/9348eedb-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
      raidz1-1                                        ONLINE       0     0     0
        gptid/82f438bf-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8d6b27be-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f66452c-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f771d80-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
      raidz1-2                                        DEGRADED     0     0     0
        gptid/8ad40550-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f548563-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        spare-2                                       DEGRADED     0     0     0
          gptid/924fd920-6a02-11ea-a922-a0369f4e18bc  FAULTED      6     4     0  too many errors
          gptid/44ad6394-6aec-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8ac32412-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
    spares
      6313981654537907299                             INUSE     was /dev/gptid/44ad6394-6aec-11ea-a922-a0369f4e18bc

errors: No known data errors

Now what? I'd like to use that successfully resilvered Spare as a permanent replacement. But in the UI, "Replace" doesn't offer up the spare as a replacement. There's no "promote" or anything option on the spare.

The command line does suggest 'zpool clear' but I'm investigating working in the UI here, so I don't want to do that. I'm interested to know what it does though - just clears the Degraded status I'm guessing?

The only other GUI option of interest was "offline" for the dead drive, so I did that. No change to the spare's status. So I onlined it, because why not? And now I have this:

Code:

  pool: DataPool0
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Mar 21 10:28:00 2020
    4.11T scanned at 3.84G/s, 2.73T issued at 2.33G/s, 21.4T total
    0 resilvered, 12.77% done, 0 days 02:16:31 to go
config:

    NAME                                              STATE     READ WRITE CKSUM
    DataPool0                                         ONLINE       0     0     0
      raidz1-0                                        ONLINE       0     0     0
        gptid/7abf8085-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8d7ccf17-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/94977190-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/9348eedb-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
      raidz1-1                                        ONLINE       0     0     0
        gptid/82f438bf-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8d6b27be-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f66452c-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f771d80-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
      raidz1-2                                        ONLINE       0     0     0
        gptid/8ad40550-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        gptid/8f548563-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
        spare-2                                       ONLINE       0     0     0
          gptid/924fd920-6a02-11ea-a922-a0369f4e18bc  ONLINE       6     4     0
          gptid/44ad6394-6aec-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8ac32412-6a02-11ea-a922-a0369f4e18bc    ONLINE       0     0     0
    spares
      6313981654537907299                             INUSE     was /dev/gptid/44ad6394-6aec-11ea-a922-a0369f4e18bc

errors: No known data errors

No longer degraded. It looks like it's resilvering onto the previously-faulted drive, which is cool and all but I don't really expect it to be successful and why is it bothering when there's a perfectly capable member disk in the spare? Ok, it's because I asked it to online the duff disk, I know, my fault :)

For some reason clear documentation for this process is hard to find - which is why I started this thread. As far as I can tell a simple cli 'zpool detach' on the failed drive should trigger the spare to belong to the vdev permanently, and since I don't really care about this resilver, here we go:

Code:

root@Sisyphus:/mnt # zpool detach DataPool0 gptid/924fd920-6a02-11ea-a922-a0369f4e18bc
root@Sisyphus:/mnt # zpool status DataPool0
  pool: DataPool0
 state: ONLINE
  scan: resilvered 428M in 0 days 00:00:07 with 0 errors on Sat Mar 21 10:56:51 2020
config:

    NAME                                            STATE     READ WRITE CKSUM
    DataPool0                                       ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/7abf8085-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8d7ccf17-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/94977190-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/9348eedb-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
      raidz1-1                                      ONLINE       0     0     0
        gptid/82f438bf-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8d6b27be-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8f66452c-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8f771d80-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
      raidz1-2                                      ONLINE       0     0     0
        gptid/8ad40550-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8f548563-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/44ad6394-6aec-11ea-a922-a0369f4e18bc  ONLINE       0     0     0
        gptid/8ac32412-6a02-11ea-a922-a0369f4e18bc  ONLINE       0     0     0

errors: No known data errors

Ok, that looks happy. Very simple action too.

Time for a bug to ask for a 'replace with spare' option to be added to the zpool status page of the GUI in this faulted+spare situation, or is it on the 'to do' list already?

 

[sretalla]

Other than adding (and maybe removing) the spare, I don't think the GUI has taken the spare features into account.

Feel free to make a feature request. I guess that the kind of folk who will use that sort of feature will be on CLI anyway.

 

[JaimieV]

Uh. I have a followup thread with more questions because spares are CLEARLY cleverer than they should be.