Volume is DEGRADED. Sufficient replicas exist - help w/ first disk failure

[Jon K]

Hi all - I've experienced my first disk failure. I can't complain, the disk has 7.2 years of on time.

So, I got the notice first through a cron job script I created and get through email:

/dev/da1 status is Passed with 0 bad sectors. Disk temperature is 40.
/dev/da2 status is Passed with 0 bad sectors. Disk temperature is 38.
/dev/da3 status is Passed with 0 bad sectors. Disk temperature is 44.
/dev/da4 status is Passed with 0 bad sectors. Disk temperature is 42.
/dev/da5 status is Passed with 0 bad sectors. Disk temperature is 43.
/dev/da6 status is Passed with 0 bad sectors. Disk temperature is 41.
/dev/da6 status is Passed with 0 bad sectors. Disk temperature is 41.
/dev/da7 status is Passed with 0 bad sectors. Disk temperature is 38.
/dev/da8 status is Passed with 0 bad sectors. Disk temperature is 43.
/dev/da9 status is Passed with 0 bad sectors. Disk temperature is 41.
/dev/da10 status is Passed with 0 bad sectors. Disk temperature is 39.
/dev/da11 status is Passed with 65535 bad sectors. Disk temperature is 39.
/dev/da12 status is Passed with bad sectors. Disk temperature is 31.
/dev/da13 status is Passed with 0 bad sectors. Disk temperature is 31.

I then logged into the UI and saw the alert: Nov. 7, 2016, 2:14 a.m. - The volume StoragePool (ZFS) state is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

Ok, all good. Strange thing is, if I run a smartctl the disk comes up passed. Here is a pastebin link of the output if interested.

So, being my first failure - how do I proceed? Here is my zpool status:

# zpool status
pool: StoragePool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 2h0m with 0 errors on Tue Nov 1 04:00:18 2016
config:

NAME STATE READ WRITE CKSUM
StoragePool DEGRADED 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/ab021123-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/abd7bff0-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/acb2c91d-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
gptid/ad959ba1-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/ae6c1740-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/af441cf6-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
gptid/b11f0d9d-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/b35a8287-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/b48f7186-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
raidz1-3 DEGRADED 0 0 0
gptid/b56dd0c7-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
gptid/b642eb2f-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0
2251441723946832699 UNAVAIL 12 64 0 was /dev/gptid/b8213b1c-472b-11e6-8b2f-d4ae5269829f
cache
gptid/b8c8ef5d-472b-11e6-8b2f-d4ae5269829f ONLINE 0 0 0


So, what are the next steps? Sorry if it's obvious. Thanks all!

 

[hugovsky]

You should replace that disk ASAP. You can try by staring to read here and remember: ALWAYS HAVE UPDATED BACKUPS.

Just because smart tells you that your disk pass the tests, doesn't mean the disk is fine.

 

[Ericloewe]

For future reference, please provide the output of smartctl -a or -x (lowercase).

The immense amount of reallocated sectors is a very bad sign - it's even overflowed the counter.

 

[Jon K]

You should replace that disk ASAP. You can try by staring to read here and remember: ALWAYS HAVE UPDATED BACKUPS.

Just because smart tells you that your disk pass the tests, doesn't mean the disk is fine.

Absolutely agree that's why my script includes bad sectors. I do have backups of all the VMs on this array.

 

[Jon K]

For future reference, please provide the output of smartctl -a or -x (lowercase).

The immense amount of reallocated sectors is a very bad sign - it's even overflowed the counter.

Yep that's the first thing I noticed. A couple hundred would be bad. The fact that it's all of the 8 byte data is troubling. No worries, though.

Here is the output of smartctl -a via pastebin.

So - how do I accurately identify the slot/etc.? And, what are the best steps to replace? Any value in retrying this disk in some manner?

 

[SweetAndLow]

You use the S/N GTE000PAJX8NKE and find the disk in your system. If you have an open slot or open sata port you should replace the disk in place. Do this by putting new disk in and in the GUI follow the steps in the manual to replace the drive. Then when the resilver is all done it will offline the old disk and you can remove it.

If you do not have an extra slot or sata port you need to offline the bad disk in the gui, shutdown system, remove old drive, put new drive in, start system and in the gui replace the drive.

 

[Ericloewe]

The fact that it's all of the 8 byte data is troubling.

Overflowing 8 bytes would mean overflowing beyond the size of any known hard drive. ;)
It's "just" 16 bits.

So - how do I accurately identify the slot/etc.?

You'll need to check the drive labels for the serial number SMART reports.
Some users keep lists of serial numbers by drive slot, and some of those users are too paranoid to trust them despite having them.
If you replace the drive while it's still connected (always recommended, if possible), you could also run a scrub and see which drive isn't being accessed or dd to the drive to mark it - assuming you have one activity LED per drive.

And, what are the best steps to replace?

Follow the manual.

Any value in retrying this disk in some manner?

I don't think so, it has just too many remapped sectors. I'd suspect some mechanical event that took out a portion of the drive's surface, which makes the whole thing a bit dubious.

 

[Ericloewe]

Oh, wait, the drive is already UNAVAIL, so might as well remove it outright.

 

[Jon K]

Oh, wait, the drive is already UNAVAIL, so might as well remove it outright.

Yep - it's unavailable in the UI so I can't offline etc. I don't suppose there's a way to flash the activity light, is there? dd won't work if the drive is unavailable I suspect. I guess I could write to the whole array and watch the LED not blink.

I have the slot labeled based on serial # but I forget how I did that.

This is in a Dell R510 12-bay 2U server. I totally forget how I mappedf serial to slot.

Ha - I remembered how I did it:

# sas2ircu 0 display

Device is a Hard disk
Enclosure # : 2
Slot # : 9
SAS Address : 500065b-3-6789-abf1
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Hitachi HUA72101
Firmware Revision : A74A
Serial No : GTE000PAJX8NKE
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

 

[Jon K]

One last question - I apologize - how do you determine whether the array needs to be shutdown for replacement? Would really like to avoid that if possible.

 

[SweetAndLow]

What is your hardware? Your motherhood, hba and chassis need to support it.

 

[Jon K]

It's a Dell R510 12-bay chassis using an a Dell PERC H200 in IT mode.

 

[Ericloewe]

One last question - I apologize - how do you determine whether the array needs to be shutdown for replacement? Would really like to avoid that if possible.

• You've verified that hot-swapping works as expected on your specific hardware
• You're absolutely sure where the drive is

 

[Jon K]

• You've verified that hot-swapping works as expected on your specific hardware
• You're absolutely sure where the drive is
• You've storage vMotion'd critical VMs to another array

FTFY :)

 

[SweetAndLow]

• You've verified that hot-swapping works as expected on your specific hardware
• You're absolutely sure where the drive is

Yes this is super important! Since you are running raid z1 you can't remove the wrong drive or you will risk that vdev completely failing.

 

[Jon K]

Yes this is super important! Since you are running raid z1 you can't remove the wrong drive or you will risk that vdev completely failing.

Yep - absolutely understand. The SAS2IRCU tool helps immensely.

Random question - I got one instance of the alert of the degraded pool today at 2:15AM. Is there any way to have it repeatedly email maybe once every 30 min or something to that effect?

 

[Stux]

You can probably Cron anything.

A better solution might've been Z2 ;)

Good luck with the replacement.

I'd consider refreshing your backup first

 

[Jon K]

You can probably Cron anything.

A better solution might've been Z2 ;)

Good luck with the replacement.

I'd consider refreshing your backup first

It bugs me that everyone's response is "Z2 would help." Help what? In 7.2 years I had a disk failure :) I didn't lose any data yet and if I do it's backed up daily. This is a pool for VM storage so if I went Z2 my performance would be worse and the redundancy might not be that much better depending. I am going to be replacing the disk tonight so we'll see how it goes :)

 

[SweetAndLow]

Performance difference will not be noticeable. Z1 use to be just fine when you built this pool when disk where only 1tb at the max. Now days you will probably have a read error during rebuild with how large disks have gotten. I have no clue how big your disks are our how well you maintain your system. Your rebuild could be just fine or it could not. Everyone's risk management is different so you don't have to listen to people just know what level of risk you are taking on.

 

[Jon K]

Performance difference will not be noticeable. Z1 use to be just fine when you built this pool when disk where only 1tb at the max. Now days you will probably have a read error during rebuild with how large disks have gotten. I have no clue how big your disks are our how well you maintain your system. Your rebuild could be just fine or it could not. Everyone's risk management is different so you don't have to listen to people just know what level of risk you are taking on.

Absolutely agree - if these were 4TB consumer SATAs. The process for this couldn't have been more simple. Yanked the failed drive out of Slot #9 in the R510, plopped a new one in, clicked "Replace", and away she goes.

I just don't think the default argument should be "RAIDZ2 would be better". That all depends on the number of disks in the vdev, their size, their speed, and whether or not consumer or enterprise.

I do have to say I am a little sad that this disk didn't hit 65k hours of on time. Oh well.

Out of curiosity, since I do have a place to put this data to rebuild the pool, what configuration would you have recommended using RAIDZ2? I have 12 disks in total. A single vdev? Two vdevs?