zpool faulted after power failure

[zoomzoom]

psu died and now I'm facing issues with the zpool

If you don't have a UPS, it's highly recommended to invest in one (preferably one with a Pure Sine Wave).

A FreeNAS server without a UPS is asking for data loss, and it's likely at least some of your data will be unrecoverable (it's entirely possible your data may never be able to be recovered). You're going to want to utilize Forum Search (@cyberjock likely has some helpful posts) to determine what needs to be done to try and salvage your pool, but due to the way in which ZFS operates, expect at least some data loss/corruption.

 

[pro lamer]

If you don't have a UPS, it's highly recommended to invest in one (preferably one with a Pure Sine Wave).

A FreeNAS server without a UPS is asking for data loss, and it's likely at least some of your data will be unrecoverable (it's entirely possible your data may never be able to be recovered). You're going to want to utilize Forum Search (@cyberjock likely has some helpful posts) to determine what needs to be done to try and salvage your pool, but due to the way in which ZFS operates, expect at least some data loss/corruption.

Are you sure? Isn't the copy on write against it... I've seen some post recently, can't find it easily but I think it was by @Arwen....

Sent from my phone

 

[Arwen]

Yes @pro lamer, ZFS was specifically designed against un-expected power loss or OS crashes. ZFS may loose data in flight, (where a transaction group write is not yet complete), but all existing data should be fine.

That said, there are hardware faults that can cause loss of existing data. So ZFS is not the end all for data longevity, backups play a part.
(We won't mention ZFS bugs that may cause data loss

 

[antipop]

I have a UPS and the Nas was plugged in it (and with the usb cable to automatically shutdown the nas in case of a power outage).

So far I wasn’t able to recover anything at all. So I’m not so sold on ZFS anymore.

 

[zoomzoom]

Are you sure? Isn't the copy on write against it...

I was referring to any data within the pool that was in use at the time of the power loss, not existing data in the pool. I should have better articulated that, as the italics in parentheses was referring to the statement before it "...it's likely at least some of your data will be unrecoverable...".

I have a UPS and the Nas was plugged in it (and with the usb cable to automatically shutdown the nas in case of a power outage).

Did you configure the UPS service via the WebAdmin? Does your UPS provide enough battery power to fully shutdown FreeNAS before it's battery is depleted? Either the UPS battery isn't sufficient to last through a FreeNAS power down, or the UPS service wasn't properly configured, as your system should have powered off without issue upon power being lost if both were properly configured.

So far I wasn’t able to recover anything at all. So I’m not so sold on ZFS any more.

ZFS is the best file system currently available across any OS due to a variety of factors, one of which is error correction; however there are some caveats that users need to be aware of, one of which is if a system loses power abruptly, it's likely any data in use at the time will be lost. @cyberjock has a great slideshow for users new to FreeNAS, which covers important points like this (can be found using Forum Search).

 

[pro lamer]

Did you configure the UPS service via the WebAdmin?

IIRC it was a PSU failure...

Sent from my phone

 

[Constantin]

Few home systems will feature dual, redundant power supplies. This is a great illustration why

• if a dedicated ZIL/SLOG device is used, it should feature power loss protection.
• backups are necessary even with Z3 or whatever pools.
• it's important to think through how you will respond to an event like this and how many layers of backup you want to implement. For example, consider the value of your time vs. the value of the data, etc. in designing a backup plan (i.e. on-site, off-site, frequency of backups, and so on).
• it's important to verify that a backup plan is working.

So sorry to hear that the data is currently unavailable and hope that can be fixed.

 

[antipop]

IIRC it was a PSU failure...

Sent from my phone

You are correct. UPS or not that would not have changed anything.

The NAS wasn’t in use when this happened so I don’t understand why I’ve lost all my data and access to the pool. Unless there is a failure of all 4 drives at the same which is what I’m trying to rule out now.

For information the drive are from different batches since 2 were replaced over time as the previous drives were starting to show errors.

 

[pro lamer]

So I've installed freenas on the laptop using a VM

Just a wild guess: have you tried booting and importing to some Linux? Disclaimer: dunno if it helps or is safe.

OTOH since the metadata is told to be corrupted... :/

Sent from my phone

 

[antipop]

I’ve tried importing on a freebsd rescue disk but I can try on a Linux system and see if that changes anything.

 

[zoomzoom]

IIRC it was a PSU failure...

My bad, I missed that in my first reading of the OP

 

[antipop]

I've installed ubuntu 18.10 and tried to import the pool. Using

Code:

sudo pool import -fFX gDisk

I found that I have a lot more disk activity than ever before which I found interesting and positive.

However after a few minutes, I received the following message

cannot import ‘gDisk: one or more devices is currently unavailable

I tried to extract the message in dmesg and I got the following once that could be of interest.

sd 3:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
sd 3:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 fc 0d a0 00 00 00 b0 00 00
print_req_error: I/O error, dev sdb, sector 16518560

sd 3:0:0:1: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driver-byte=DRIVER_OK
sd 3:0:0:1: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 fb 8a 10 00 00 00 b0 00 00
print_req_error: I/O error, dev sdc, sector 16484880

sd 3:0:0:2: [sdd] Unaligned partial completion (resid=40947, sector_sz=512)
sd 3:0:0:2: [sdd] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:2: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 09 ed 70 00 00 01 00 00 00 00
print_req_error: I/O error, dev sdd, sector 17427824

sd 3:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 01 09 62 00 00 00 00 b0 00 00
print_req_error: I/O error, dev sde, sector 17392128

sd 3:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:0: [sdb] Sense note available

sd 3:0:0:1: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:1: [sdc] Sense note available

sd 3:0:0:2: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:2: [sdd] Sense note available

sd 3:0:0:3: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:3: [sde] Sense note available


sd 3:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:0: [sdb] Sense note available
sd 3:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B)
sd 3:0:0:0: [sdb] 4096-byte physical blocks

sd 3:0:0:1: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:1: [sdc] Sense note available
sd 3:0:0:1: [sdc] 0 512-byte logical blocks: (0 B/0 B)
sd 3:0:0:1: [sdc] 4096-byte physical blocks

sd 3:0:0:2: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:2: [sdd] Sense note available
sd 3:0:0:2: [sdd] 0 512-byte logical blocks: (0 B/0 B)
sd 3:0:0:2: [sdd] 4096-byte physical blocks

sd 3:0:0:3: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:3: [sde] Sense note available
sd 3:0:0:3: [sde] 0 512-byte logical blocks: (0 B/0 B)
sd 3:0:0:3: [sde] 4096-byte physical blocks


sd 3:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 fc 0e 50 00 00 00 b0 00 00
print_req_error: I/O error, dev sdb, sector 16518736

sd 3:0:0:1: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driver-byte=DRIVER_OK
sd 3:0:0:1: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 fb 8a c0 00 00 00 b0 00 00
print_req_error: I/O error, dev sdc, sector 16485056

sd 3:0:0:2: [sdd] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:2: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 09 ee 78 00 00 01 00 00 00 00
print_req_error: I/O error, dev sdd, sector 17428088

sd 3:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 3:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 01 09 62 b0 00 00 00 b0 00 00
print_req_error: I/O error, dev sde, sector 17392304

Any idea what that could mean?