Power outage pool state UNAVAIL

[poohbear]

Had a power outage and the pool is no more. All the drives say cannot open but they are there. Can anyone help with getting this back online?

Build: FreeNAS-8.2.0-RELEASE-p1-x64 (r11950)

Code:
---------
zpool status
pool: lun0
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
lun0 UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 insufficient replicas
replacing UNAVAIL 0 0 0 insufficient replicas
1106485570164544559 OFFLINE 0 0 0 was /dev/ada0/old
ada0 UNAVAIL 0 0 0 cannot open
ada0 UNAVAIL 0 0 0 cannot open
ada1 UNAVAIL 0 0 0 cannot open
ada2 UNAVAIL 0 0 0 cannot open

ls -l /dev/ad*
crw-r----- 1 root operator 0, 92 Jan 14 14:16 /dev/ada0
crw-r----- 1 root operator 0, 102 Jan 14 14:16 /dev/ada0p1
crw-r----- 1 root operator 0, 103 Jan 14 14:16 /dev/ada0p2
crw-r----- 1 root operator 0, 93 Jan 14 14:16 /dev/ada1
crw-r----- 1 root operator 0, 104 Jan 14 14:16 /dev/ada1p1
crw-r----- 1 root operator 0, 105 Jan 14 14:16 /dev/ada1p2
crw-r----- 1 root operator 0, 94 Jan 14 14:16 /dev/ada2
crw-r----- 1 root operator 0, 106 Jan 14 14:16 /dev/ada2p1
crw-r----- 1 root operator 0, 107 Jan 14 14:16 /dev/ada2p2
crw-r----- 1 root operator 0, 95 Jan 14 14:16 /dev/ada3
crw-r----- 1 root operator 0, 108 Jan 14 14:16 /dev/ada3p1
crw-r----- 1 root operator 0, 109 Jan 14 14:16 /dev/ada3p2

zdb -l /dev/ada0
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

camcontrol devlist
<WDC WD20EARX-07PASB0 51.0AB51> at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD20EARS-00MVWB0 50.0AB50> at scbus1 target 0 lun 0 (pass1,ada1)
<WDC WD20EARS-00MVWB0 50.0AB50> at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD20EARS-00MVWB0 50.0AB50> at scbus3 target 0 lun 0 (pass3,ada3)

 

[Stephens]

Do you have a UPS?

 

[poohbear]

A ups was not attached to the freenas box

 

[William Grzybowski]

Well, I tend to think people usually hide details for whatever reason :)

It is hard to believe that is all that happened. ZFS thinks the metadata were in the whole disks, yet you have a partition table over those disks now?

Something is not right in this history.

 

[cyberjock]

A ups was not attached to the freenas box

Whoops. Guess you'll put an UPS on all of your servers from now on like a good admin should. Huh?

It's not like it hasn't been recommended that an UPS be used on a server for decades...nor has it ever been discussed and discussed on this forum either.

Well, I tend to think people usually hide details for whatever reason :)

It is hard to believe that is all that happened. ZFS thinks the metadata were in the whole disks, yet you have a partition table over those disks now?

Something is not right in this history.

Totally agree. I'm thinking he did something like install Windows on the machine then went back to use the disks on FreeNAS.

 

[poohbear]

yup a ups would have been a good idea. This is just a test server not a production box. I figure it should have been able to take a power outage. A ups isn't going to protect a failed motherboard or controller failing so I'd be in the same boat. Anyone one actually have an idea before I wipe it and toss freenas?

 

[jgreco]

Sure, show us the output of "gpart list".

 

[poohbear]

~# gpart list
Geom name: da0
state: OK
fwheads: 255
fwsectors: 63
last: 7913429
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
Mediasize: 988291584 (943M)
Sectorsize: 512
Mode: r0w0e0
rawtype: 165
length: 988291584
offset: 32256
type: freebsd
index: 1
end: 1930319
start: 63
2. Name: da0s2
Mediasize: 988291584 (943M)
Sectorsize: 512
Mode: r1w0e1
attrib: active
rawtype: 165
length: 988291584
offset: 988356096
type: freebsd
index: 2
end: 3860639
start: 1930383
3. Name: da0s3
Mediasize: 1548288 (1.5M)
Sectorsize: 512
Mode: r0w0e0
rawtype: 165
length: 1548288
offset: 1976647680
type: freebsd
index: 3
end: 3863663
start: 3860640
4. Name: da0s4
Mediasize: 21159936 (20M)
Sectorsize: 512
Mode: r1w1e2
rawtype: 165
length: 21159936
offset: 1978195968
type: freebsd
index: 4
end: 3904991
start: 3863664
Consumers:
1. Name: da0
Mediasize: 4051697664 (3.8G)
Sectorsize: 512
Mode: r2w1e4

Geom name: da0s1
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
Mediasize: 988283392 (943M)
Sectorsize: 512
Mode: r0w0e0
rawtype: 0
length: 988283392
offset: 8192
type: !0
index: 1
end: 1930256
start: 16
Consumers:
1. Name: da0s1
Mediasize: 988291584 (943M)
Sectorsize: 512
Mode: r0w0e0

Geom name: da0s2
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s2a
Mediasize: 988283392 (943M)
Sectorsize: 512
Mode: r1w0e1
rawtype: 0
length: 988283392
offset: 8192
type: !0
index: 1
end: 1930256
start: 16
Consumers:
1. Name: da0s2
Mediasize: 988291584 (943M)
Sectorsize: 512
Mode: r1w0e1

Geom name: ada0
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e0
rawuuid: be77236e-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada0p2
Mediasize: 1998251367936 (1.8T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: be85bfe4-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251367936
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029134
start: 4194432
Consumers:
1. Name: ada0
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1

Geom name: ada1
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e0
rawuuid: bf1cc7f2-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada1p2
Mediasize: 1998251367936 (1.8T)
Sectorsize: 512
Mode: r0w0e0
rawuuid: bf291c90-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251367936
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029134
start: 4194432
Consumers:
1. Name: ada1
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e1

Geom name: ada2
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e0
rawuuid: bfbf8d3a-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada2p2
Mediasize: 1998251367936 (1.8T)
Sectorsize: 512
Mode: r0w0e0
rawuuid: bfccac1d-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251367936
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029134
start: 4194432
Consumers:
1. Name: ada2
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e1

Geom name: ada3
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e0
rawuuid: c060be16-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada3p2
Mediasize: 1998251367936 (1.8T)
Sectorsize: 512
Mode: r0w0e0
rawuuid: c06e13d4-2626-11e2-b8a8-00012e335ca9
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251367936
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029134
start: 4194432
Consumers:
1. Name: ada3
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e1

 

[poohbear]

used a freebsd 9.1 livecd and was able to import the pool and access data.

 

[cyberjock]

WOW. That's so bizarre. I pretty much wrote off your data as lost for good before your last 2 posts. I wonder why it works with FreeBSD 9.1 but not FreeNAS 8.3. I'd really like to know why...

 

[poohbear]

yup odd stuff. No errors nothing export pool, auto import from gui everything back to normal using freenas 8.3 now. Doesn't make me feel confident about freenas. I'm wondering if there was some sort of corruption in the sqlite database the gui uses to track stuff and it was giving false info about the pool status.