Did I just screwed up my zpool?

[sac75578]

Hi, I had (4) 3tb seagate raidz with 1 drive for redundancy. 1 of the drives failed and was no longer detectable and other drives had errors. I turned off the pc and replaced what I thought was the failed drive. I plugged that drive into an external Sata drive reader on a Windows machine. I didn't do anything except turning it on. Meanwhile I turn on the Freenas machine and was unable to mount the zpool. I realized the mistake and tried to put the old drive back but the zpool status now reports as unknown. 3 out of the 4 drives are detectable but 2 are unavailable. What should I do now? Thanks

 

[DrKK]

Hi, I had (4) 3tb seagate raidz

I can already answer, "yes, you screwed up your pool" right there.

 

[DrKK]

In all seriousness,

So let me get this straight. A drive failed. You thought it was drive X. You pull out drive X (and then mount it on an external SATA bridge on a windows box). While drive X is pulled, you replace drive X with drive Z. You decided drive X was actually fine, and drive Y is jacked. So you put drive X back in the NAS, and now you can't mount.

Is that correct? NOthing was officially resilvered, etc?

 

[sac75578]

Yes that's exactly what happened.

In all seriousness,

So let me get this straight. A drive failed. You thought it was drive X. You pull out drive X (and then mount it on an external SATA bridge on a windows box). While drive X is pulled, you replace drive X with drive Z. You decided drive X was actually fine, and drive Y is jacked. So you put drive X back in the NAS, and now you can't mount.

Is that correct? NOthing was officially resilvered, etc?

 

[DrKK]

Yes that's exactly what happened.

Let's call the original drives V, W, X, and Y. And the new drive is Z. X is the drive you thought was broken. Y is the drive actually broken.

When you have V, W, X, Y back in the box now, you cannot get a mount. Correct?

If so, let's see the result of "camcontrol devlist" (in 'code' tags), and the result of "zpool import"

 

[sac75578]

here's the results of the commands:

Code:

[root@freenas ~]# camcontrol devlist                                            
<ST3000DM001-9YN166 CC9F>          at scbus0 target 0 lun 0 (ada0,pass0)        
<ST3000DM001-9YN166 CC9E>          at scbus2 target 0 lun 0 (ada1,pass1)        
<ST3000DM001-9YN166 CC9F>          at scbus3 target 0 lun 0 (ada2,pass2)        
< Patriot Memory PMAP>             at scbus5 target 0 lun 0 (da0,pass3)  
[root@freenas ~]# zpool import                                                  
   pool: home                                                                  
     id: 16221184243623947772                                                  
  state: UNAVAIL                                                                
 status: One or more devices are missing from the system.                      
 action: The pool cannot be imported. Attach the missing                        
        devices and try again.                                                  
   see: http://illumos.org/msg/ZFS-8000-3C                                      
 config:                                                                        
                                                                               
        home                                            UNAVAIL  insufficient re
plicas                                                                          
          raidz1-0                                      UNAVAIL  insufficient re
plicas                                                                          
            14071875661356975025                        UNAVAIL  cannot open    
            gptid/1295d955-4262-11e2-af98-902b3400f032  ONLINE                  
            gptid/1308689e-4262-11e2-af98-902b3400f032  ONLINE                  
            17012602856798749812                        UNAVAIL  cannot open  

 

[Ericloewe]

Try removing all drives except for X. Tell us what zpool import says.

 

[sac75578]

When I just leave the X drive in, zpool import does nothing.

Code:

[root@freenas ~]# zpool import
[root@freenas ~]#

Try removing all drives except for X. Tell us what zpool import says.

 

[Ericloewe]

So X is hosed as well, somehow. Doesn't look good. At this point I hope it's a PEBKAC.

Try connecting all the drives (all of them) and run zpool import.

 

[sac75578]

A few post above has the zpool import output with all drives connected.

So X is hosed as well, somehow. Doesn't look good. At this point I hope it's a PEBKAC.

Try connecting all the drives (all of them) and run zpool import.

 

[Ericloewe]

A few post above has the zpool import output with all drives connected.

Including the new drive? Camcontrol devlist only shows three, so you're missing one more than expected (assuming the original failed drive is completely dead).

You should also ensure that it's not a cabling issue. Make sure all drives are being powered and try replacing the SATA cables.

 

[sac75578]

The camcontrol devlist command has all 4 original drive plugged in the pc including the undetectable one. I've replaced all the SATA cables and power supply and tried using IDE mode and still not being detected. I'll try to see if it will get detected on another pc (just bootup, no windows).

Including the new drive? Camcontrol devlist only shows three, so you're missing one more than expected (assuming the original failed drive is completely dead).

You should also ensure that it's not a cabling issue. Make sure all drives are being powered and try replacing the SATA cables.

 

[Ericloewe]

tried using IDE mode

Don't bother.

The camcontrol devlist command has all 4 original drive plugged in

Plug in the "new" drive, too.

 

[sac75578]

Nothing was done to the new drive when I had it in the Freenas, i plugged that back in to the Windows drive and the files were originally there are still there.

I tried plugging in the 4 original drives on a new PC and all 4 were detected by the PC on startup. However once Freenas boots, one of them becomes undetectable. I think it was stalling when it was waiting for ahci reset then timed out. I reboot and the drive becomes unavailable again until I turn off PC and turn it back on. How can I make the drive stay detectable for Freenas?

 

[Robert Trevellyan]

How can I make the drive stay detectable for Freenas?

You could try cloning it with GNU ddrescue.

 

[Robert Trevellyan]

1 of the drives failed and was no longer detectable and other drives had errors

How did you determine this?
Did you have regular SMART tests and SMART checks configured, with email notifications working?

ST3000DM001-9YN166

My guess is your pool was already degraded (no redundancy), so when the 2nd drive failed, it was game over.

 

[sac75578]

I individually plug in each hard drive and made sure it detected in the bios and then checked the UI in Freenas to see if it still sees the drive. One of them does not stay detected. Could the 2nd drive failed right when I tried to replace the 1st bad drive? Am I that unlucky?

How did you determine this?
Did you have regular SMART tests and SMART checks configured, with email notifications working?

My guess is your pool was already degraded (no redundancy), so when the 2nd drive failed, it was game over.

 

[Bidule0hm]

Could the 2nd drive failed right when I tried to replace the 1st bad drive

It can. Did you burned it in before using it?

 

[sac75578]

Smart test sent emails with undetectable sector errors and one drive failure.

How did you determine this?
Did you have regular SMART tests and SMART checks configured, with email notifications working?

My guess is your pool was already degraded (no redundancy), so when the 2nd drive failed, it was game over.

 

[sac75578]

It can. Did you burned it in before using it?

I'm not sure what you mean by burned it in. The 2nd drive I'm referring to is not the new drive. Im referring to a 2nd drive in the original 4 failed right when I tried to replace the 1st bad drive.