One or more devices are faulted in response to IO failures. But tank and drives flagged as ONLINE

[Mattias Larsson]

Hello everyone.

I ran into a little problem on my NAS the other day. For some reason it hang and refused to finish booting or letting me in. Finally I managed to sneak in during boot just after sshd but before it became unresponsive. It turns out that it has "disabled" my only pool (except for freenas-boot that is) but the pool and all 6 of it's disks where in ONLINE status. This is what zpool status shows me:

Code:

status: One or more devices are faulted in response to IO failures.

NAME  STATE  READ WRITE CKSUM
tank  ONLINE  0  4  0
  mirror-0  ONLINE  0  0  0
  gptid/577aeadd-d136-11e5-9e4c-0cc47a6cb6ea  ONLINE  0  0  0
  gptid/e1a4ec64-0707-11e6-8e7f-0cc47a6cb6ea  ONLINE  0  0  0
  mirror-1  ONLINE  0  0  0
  gptid/600be257-d4c6-11e5-a781-0cc47a6cb6ea  ONLINE  0  0  0
  gptid/60b14524-d4c6-11e5-a781-0cc47a6cb6ea  ONLINE  0  0  0
  mirror-2  ONLINE  0  0  0
  gptid/a03db210-4557-11e7-8c90-0cc47a6cb6ea  ONLINE  0  0  0
  gptid/a0e085b5-4557-11e7-8c90-0cc47a6cb6ea  ONLINE  0  0  0

I ran long smart-tests on all drives and they all reported OK status. I have replaced the cables and tried switching from the SAS-ports on my motherboard to the SATA ones (SuperMicro X10SL7-F). I can't find any indication of errors in any logfile either so I'm starting to wonder what the heck could be wrong.

Is it possible the filesystem itself is somehow corrupt? I have had tank for several years but without underlaying errors it should happen right?

Any ideas would be welcomed, either possible reason or methods to troubleshoot. I'm not the best at these kinds of problems...

// Mattias

 

[Jailer]

Is it possible the filesystem itself is somehow corrupt? I have had tank for several years but without underlaying errors it should happen right?

We wont know for sure since you cropped the output of zpool status. Post the entire output along with a complete hardware list and maybe we can help.

 

[Mattias Larsson]

Unfortunatly I don't have the full output of zpool status anymore, forgot to save the output and I have done some things since then.

I have since removed 3 of the 6 6TB WD-RED drives, one from each mirror and at least for the moment the pool seems to behave so it might very well have been one of the drives acting up. A little strange that Smart didn't pick up any problems but I don't know nearly enough about Smart to say if that's normal or not.

The next logical step I guess would be to reattach one drive at a time and see how it goes.

As for my hardware I'm running:

FreeNAS-11.0-U4
Supermicro X10SL7-F
Xeon E3-1230v3
32GB (4x8GB Crucial ECC DDR3)
6x6TB WD Red
Corsair AX860

 

[Jailer]

No thats not the next logical step. Youve just removed all redunency from your pool. If you pulled one of the wrong drives and you experience another drive problem you will lose your pool.

This is a perfect example of what NOT to do if your pool is degraded.

 

[Mattias Larsson]

Yes, I'm aware of that. Luckily none of the data is all that critical, a shame to loose it but not devestating. At the moment I'm running with one drive per pool so yea, should other one fail I'm screwed. So far the pool is working, degraded ofc.