ZPool IO Error: unable to import: RAIDZ2 6 x 3TB

[dassiq]

Hi All

Need some help. This happened after a power failure . I am unable to import one of my Pools.

FreeNAS-9.3-STABLE-201605170422

I am unable to import the pool 'FNV4'.

6 x 3TB drives in RAIDZ2
All drives are deemed fine, with no smart errors.
Please see results from the zpool commands below.

Doing some basic Google search points to metadata issue.

Code:

[root@FNN ~]# zpool import                                                                                                         
   pool: FNV4                                                                                                                       
     id: 13913909973389052943                                                                                                       
  state: ONLINE                                                                                                                     
action: The pool can be imported using its name or numeric identifier.                                                             
config:                                                                                                                           
                                                                                                                                   
        FNV4                                            ONLINE                                                                     
          raidz2-0                                      ONLINE                                                                     
            gptid/894df5cc-8b5d-11e5-88c8-003048d1e578  ONLINE                                                                     
            gptid/89b56663-8b5d-11e5-88c8-003048d1e578  ONLINE                                                                     
            gptid/8a1348b9-8b5d-11e5-88c8-003048d1e578  ONLINE                                                                     
            gptid/8a7242eb-8b5d-11e5-88c8-003048d1e578  ONLINE                                                                     
            gptid/8ad11c1c-8b5d-11e5-88c8-003048d1e578  ONLINE                                                                     
            gptid/a5ca7dad-9646-11e5-af95-003048d1e578  ONLINE                                                                     
[root@FNN ~]#        

Code:

[root@FNN ~]# zpool import -f FNV4                                                                                                 
cannot import 'FNV4': I/O error                                                                                                     
        Destroy and re-create the pool from                                                                                         
        a backup source.                   

 

[rs225]

You can also try zpool import -f -F FNV4

What disk controller are you using, and what type of disks?

 

[dassiq]

There are 3 more pools (6 x 3TB each) on the same server which are working fine. All 4 pools have been working great for a year or so , until this power failure.

All pools are on one controller adaptec 31605 SAS , No BBU (which probably caused this issue)

Drives in FNV4 = 2 x Seagates , 4 x Hitachi all SATA 3TB , each drive is configured as an independent 1 drive array in the SAS controller.

Code:

[root@FNN ~]# zpool import -f -F FNV4                                                                                               
cannot import 'FNV4': I/O error                                                                                                     
        Destroy and re-create the pool from                                                                                         
        a backup source.                                                                                                           
[root@FNN ~]#

 

[Robert Trevellyan]

each drive is configured as an independent 1 drive array in the SAS controller

Then you are probably not going to be able to recover the pool.

 

[dassiq]

Why is that ? FreeNAS sees the drive as a single drive and does its sofware RAIDZ2

 

[Robert Trevellyan]

Yes, but you've still put a hardware RAID controller between ZFS and the drives. That's a recipe for data loss, because the state of non-volatile storage reported to ZFS by the controller is not necessarily the actual state of the physical disks. It will probably eat your other pools too at some point. The correct approach is to use a proper HBA, or a controller that has been flashed to IT mode.

 

[dassiq]

I am hoping we can do some thing here. If there was data that was supposed to be written as per ZFS and was not actually written, I should be able to scan file system and figure out where the problem is

 

[Robert Trevellyan]

I should be able to scan file system and figure out where the problem is

Unfortunately there is no equivalent to the fsck type tools available with other filesystems. ZFS maintains itself continuously, but it can't do that properly on the wrong hardware.

The message you're seeing is unambiguous.

cannot import 'FNV4': I/O error
Destroy and re-create the pool from a backup source.

"I/O Error", not, "device missing" or "degraded" or "faulted" or a wall of text followed by a reboot.

For the record, I'll list some of the things that sometimes help with a corrupted pool. Maybe you'll get lucky, but I doubt it. I probably sound mean, but I'm being blunt because your resources are probably better spent on recovering from backup <insert>and replacing your disk controller</insert> than trying to resurrect the pool.

Here's my basic list. There may be other things more expert folks know how to do that I'm not familiar with.

• importing read-only
• recovery mode import (-F)
• adding more RAM (if the system crashes during import)

Check the documentation for details of the zpool import options.

 

[rs225]

My guess is the controller was doing caching. Most likely the disk labels are inconsistent. If they are manually made consistent, the pool could probably be accessed. Unless it was under high write load at the time, in which case it may simply be scrambled.

Not withstanding that, it still seems like too many pools become inaccessible after power failure. Not all of them have the caching risk, so what is going on?

 

[SweetAndLow]

Another raid card destroys a pool. zfs wrote some data => your raid card lied and said yes i wrote that to disk => power loss => raid card didn't actually write it to disk so now zfs has no clue what is on the disk.

 

[dassiq]

I am going to try few things and report back.

I am going to dig deep and look at using zdb , work with uberblocks and Tsg

 

[dassiq]

Does any body know what does the output of the following command means in context of my problem

Code:

zdb -e -D FNV4                                                                                                       
All DDTs are empty                                                                                                                 
space map refcount mismatch: expected 162 != actual 131   

 

[SweetAndLow]

Space map is corrupt or something. If you figure this out make sure to post your results.

Sent from my Nexus 5X using Tapatalk

 

[rs225]

You should try importing in readonly mode.

 

[dassiq]

There were some drives on my other pools (FNV1 and FNV2)) that were showing GPT corrupt . So using Gpart recover command I fixed those.

After fixing I reran the the above zdb command for FNV4, and now I am getting more info

Code:

[root@FNN] ~# zdb -e -d FNV4
Dataset mos [META], ID 0, cr_txg 4, 269M, 204 objects
Dataset FNV4/winshare4 [ZPL], ID 42, cr_txg 37, 10.5T, 2573172 objects
Dataset FNV4 [ZPL], ID 21, cr_txg 1, 192K, 9 objects
Verified large_blocks feature refcount is correct (0)
space map refcount mismatch: expected 162 != actual 131