pool metadata corrupted

sparenton · Nov 12, 2012

Hi everyone,

This is my very first post here, and i deeply wish i wouldn't have to... I've just arrived in my company and have to manage a freenas 8.0 with 3x2TB HD. So far so good, except that this morning nobody reached the nas... when i type
zpool status -v, i have this :

pool: Vol0
state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from a backup source.
see: http://www.sun.com/msg/ZFS-8000-72
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Vol0 FAULTED 0 0 1 corrupted data
raidz1 ONLINE 0 0 6
gpt/ada0 ONLINE 0 0 0
gpt/ada1 ONLINE 0 0 0
gpt/ada2 ONLINE 0 0 0

What i've read regarding this kind of error does not please me at all. I've read things, downloaded the mfsbsd iso to try things, and though i don't have much hope in the success of it all, i'm still asking you if you have clues in order to try to solve the most data as possible.

I don't have any snapshot (it wasn't configured), i don't know yet if there is external backup... so any info is warmly welcom ;)

regards
Stephane

Stephens · Nov 12, 2012

It sucks when you're learning on a production system, so I feel for you. I don't know what you've read and what you haven, but...
http://docs.oracle.com/cd/E19253-01/819-5461/gbctt/index.html

Do you use a UPS? Do you do ANY backups? (RAID isn't backup.)

I sure hope someone can help you because you're right it doesn't look good.

cyberjock · Nov 12, 2012

Yeah... I really hope you have backups. You just might be testing how well those have been made. Judging from the fact that you are new to the company and having this problem I'm guessing backups were maintained about as well as the server was maintained. But, you never know.

There's some commands you can use to force the zpool to import, but I'll leave it to someone more knowledgable than I to try to talk you through it. But if you have backups of anything, I'd start looking at what it will take to restore those. If your company plans to continue to use FreeNAS I'd also read the manual so you are a little less lost.

Edit: I think I should add that the problem you are experiencing doesn't appear to be a problem with FreeNAS itself, so replacing FreeNAS with another OS won't likely correct the problem.

paleoN · Nov 12, 2012

sparenton said:
This is my very first post here, and i deeply wish i wouldn't have to... I've just arrived in my company and have to manage a freenas 8.0 with 3x2TB HD.

First upgrade to 8.3. It has the much newer ZFS code and is better with recovery.

Then from a SSH session as root the output of:

Code:

zpool import

Throw the output in some [code][/code] tags. This will preserve the formatting and keep my eyes from crossing.

I wouldn't mind seeing the SMART info for those 3 drives:

Code:

smartctl -q noserial -a /dev/ada0

smartctl -q noserial -a /dev/ada1

smartctl -q noserial -a /dev/ada2

You can try the following import, you will lose some data IF it works:

Code:

zpool import -fF -R /mnt Vol0

With pool metadata corruption there is a fair chance you will need backups.

sparenton · Nov 13, 2012

paleoN said:
First upgrade to 8.3. It has the much newer ZFS code and is better with recovery.

I've already tried this, but the upgrade failed on the DB upgrade. So i did go back to a 8.0.3

paleoN said:

Then from a SSH session as root the output of:

Code:

zpool import

Throw the output in some [code][/code] tags. This will preserve the formatting and keep my eyes from crossing.

I wouldn't mind seeing the SMART info for those 3 drives:

Code:

smartctl -q noserial -a /dev/ada0

smartctl -q noserial -a /dev/ada1

smartctl -q noserial -a /dev/ada2

Code:

smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p6 i386] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (Adv. Format)
Device Model:     ST2000DL003-9VT166
Firmware Version: CC32
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Nov 13 09:05:11 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  612) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30b7) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       1170032
  3 Spin_Up_Time            0x0003   094   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   030    Pre-fail  Always       -       66079762
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7318
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       34
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   056   045    Old_age   Always       -       36 (Min/Max 35/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       27
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       34
194 Temperature_Celsius     0x0022   036   044   000    Old_age   Always       -       36 (0 21 0 0)
195 Hardware_ECC_Recovered  0x001a   032   005   000    Old_age   Always       -       1170032
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       127371550137495
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3637737287
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1876700467

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Aborted by host               90%      7297         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p6 i386] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (Adv. Format)
Device Model:     ST2000DL003-9VT166
Firmware Version: CC32
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Nov 13 09:06:07 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  602) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30b7) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   102   099   006    Pre-fail  Always       -       1086136
  3 Spin_Up_Time            0x0003   094   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   030    Pre-fail  Always       -       65540850
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7388
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       33
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   097   000    Old_age   Always       -       3
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   068   060   045    Old_age   Always       -       32 (Min/Max 32/34)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       26
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       33
194 Temperature_Celsius     0x0022   032   040   000    Old_age   Always       -       32 (0 20 0 0)
195 Hardware_ECC_Recovered  0x001a   009   005   000    Old_age   Always       -       1086136
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       119223997177053
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       750112055
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2906192014

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p6 i386] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (Adv. Format)
Device Model:     ST2000DL003-9VT166
Firmware Version: CC32
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Nov 13 09:07:16 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  612) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30b7) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   108   099   006    Pre-fail  Always       -       1079488
  3 Spin_Up_Time            0x0003   094   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   030    Pre-fail  Always       -       66532525
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7358
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       33
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   057   045    Old_age   Always       -       36 (Min/Max 32/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       27
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       33
194 Temperature_Celsius     0x0022   036   043   000    Old_age   Always       -       36 (0 19 0 0)
195 Hardware_ECC_Recovered  0x001a   023   005   000    Old_age   Always       -       1079488
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       271802710367422
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1222303989
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2473116856

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]




SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

You can try the following import, you will lose some data IF it works:

Code:

zpool import -fF -R /mnt Vol0

With pool metadata corruption there is a fair chance you will need backups.

Ok, i'll test this all today... Unfortunately, i guess that there was no full backup... there is some partial backup here and there but nothing to be sure we can start from scratch with freeNas :(

Thanks for your time :)

paleoN · Nov 13, 2012

sparenton said:
I've already tried this, but the upgrade failed on the DB upgrade. So i did go back to a 8.0.3

Then don't upgrade the DB and do a fresh install on a new USB stick.

cyberjock · Nov 13, 2012

Using 8.0.3 is a bad idea in a business environment. There was a security issue that was fixed in the last 8.0.4 release.

I agree with paleoN. If you can't upgrade the DB, then you have the fantastic luxury of setting up the database again :(

sparenton · Nov 14, 2012

paleoN said:
Then don't upgrade the DB and do a fresh install on a new USB stick.

I did it. I've Installed freenas 8.3 on a stick, booted on it. I did a zpool import -fF Vol0.
The system answered me that the i may loss 3 seconds (which is way more acceptable than 1 year ;) ). The import worked successfully.

I need to be carefull now, what should I do ? zpool scrub ?

jgreco · Nov 14, 2012

My suggestion would be to BACK EVERYTHING UP BEFORE YOU DO ANOTHER THING. That's not so much because we're expecting anything to go wrong, but because this is the real world and anything that can go wrong does go wrong - but often only when you've not prepared for it. If you make a backup, you'll probably not need it. Do you have some reasonable options to do this?

paleoN · Nov 14, 2012

sparenton said:
The import worked successfully.

Excellent.

jgreco said:
My suggestion would be to BACK EVERYTHING UP BEFORE YOU DO ANOTHER THING.

+1

If you don't have some reasonable options go buy some.

sparenton · Nov 16, 2012

paleoN said:
Excellent.

+1

If you don't have some reasonable options go buy some.

I've bought enough to build a replicated Nas and, first of all, backup the disks. All these should be here on monday or tuesday....

cyberjock · Nov 16, 2012

It's really very unfortunate, but I'm betting the person that approves of the money for IT stuff wasn't approving of the money for the backups until he was in a situation where he saw first hand what it was like to almost lose everything.. Suddenly he'll spend the money now that he's seen how important backups are. I've seen that all too often, and often too late. At least you were able to get your data back.

sparenton · Nov 16, 2012

noobsauce80 said:
It's really very unfortunate, but I'm betting the person that approves of the money for IT stuff wasn't approving of the money for the backups until he was in a situation where he saw first hand what it was like to almost lose everything.. Suddenly he'll spend the money now that he's seen how important backups are. I've seen that all too often, and often too late. At least you were able to get your data back.

Well, there's a misunderstanding... i did not have my data back so far.... all i have is a zpool import -fF Vol0 that was successful (so it seems), but didn't get may volumes up and running. I've read about scrub or something and don't know what to do in order to get the volumes again in sight and useable... autoimport doesn't show anything...

sparenton · Nov 29, 2012

HI every one - Good news, my datas are all back again... I've spent time to backup the disks i had in my nas and put thoses backup disks in a brand new machine that is meant to back-up my first nas. I put up a fresh install of freenas and all went up after a import. So everythng is ok now... the thread is closed

Important Announcement for The TrueNAS Community.

pool metadata corrupted

sparenton

Dabbler

Stephens

Patron

cyberjock

Inactive Account

paleoN

Wizard

sparenton

Dabbler

paleoN

Wizard

cyberjock

Inactive Account

sparenton

Dabbler

jgreco

Resident Grinch

paleoN

Wizard

sparenton

Dabbler

cyberjock

Inactive Account

sparenton

Dabbler

sparenton

Dabbler

Similar threads