RAID array destroyed

[Dotball]

Hello all,

Firstly, sorry for the long post, I have tried to include everything I thought you may need.

My RAID array has fallen off the system and I need to double check on the best way to rebuild it hopefully without losing any data.

I cannot put my finger on what happened but I shut down the server, disconnected the power and did some maintenance on the UPS. Reconnected everything and when I turned it back on my pool was offline. I did a backup (belatedly) of the current system state and uploaded a previous config from about two weeks ago but that did change anything with the pool. Also, I have no dashboard showing on the main page.

Previously with SCSI RAID I have been able to recreate the data set with the same parameters and have the data available as if there had been no issue but I am new to ZFS and want to confirm this is the best way to go. FTR the majority of the data is photos and home videos which I have no valid off server backup of.

System
FreeNAS-11.3-U3.2
6 * 3TB = 15TB RAIDZ1
Dell H700 hardware RAID card running 6 virtual drives all being recognized 100% at boot

Drive info (sorry, cannot work out how to do the code thing correctly)

Code:

<icode>
root@pantry:~ # camcontrol devlist
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 0 lun 0 (pass0)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 1 lun 0 (pass1)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 2 lun 0 (pass2)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 3 lun 0 (pass3)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 4 lun 0 (pass4)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 5 lun 0 (pass5)
<DP BACKPLANE 1.07>                at scbus0 target 32 lun 0 (pass6,ses0)
<TEAC DVD-ROM DV28SV D.0K>         at scbus1 target 0 lun 0 (pass7,cd0)
<ATA SAMSUNG SSD PM80 4D1Q>        at scbus4 target 0 lun 0 (pass8,da0)
</icode>

<icode>
root@pantry:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:04:03 with 0 errors on Sat Jul 25 03:49:03 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors
<i/code>

<icode>
root@pantry:~ # zpool import Vault
cannot import 'Vault': I/O error
        Destroy and re-create the pool from
        a backup source.
</icode>

Questions
Should I stay on the config from two weeks ago or revert to the one I created once I found the fault?
Am I correct in being able to rebuild the array with the same parameters to be able to access my data?
Is there a better way of going forward on this?

If you have any questions or need more information please feel free to contact me.

I appreciate any assistance I can get on this as (obviously) I would really like to get this back online.

Thank you
Cheers,

 

[Dotball]

Additional information

Code:

root@pantry:~ # zpool import
   pool: Vault
     id: 11793493121225777237
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-72
 config:

        Vault                                           FAULTED  corrupted data
          raidz1-0                                      ONLINE
            gptid/2c686939-c455-11e6-9fcc-b8ac6f970a73  ONLINE
            gptid/2f044311-c455-11e6-9fcc-b8ac6f970a73  ONLINE
            gptid/319f55cd-c455-11e6-9fcc-b8ac6f970a73  ONLINE
            gptid/34331dfc-c455-11e6-9fcc-b8ac6f970a73  ONLINE
            gptid/34c5a50f-c455-11e6-9fcc-b8ac6f970a73  ONLINE
            gptid/37a638ba-c455-11e6-9fcc-b8ac6f970a73  ONLINE
root@pantry:~ #

 

[sretalla]

Am I correct in being able to rebuild the array with the same parameters to be able to access my data?

I don't think that's a good idea as it seems you have metadata corruption, not partition problems.

Dell H700 hardware RAID card running 6 virtual drives

This is not a supported setup. You need to either run the card with IT firmware (HBA mode) or get an HBA if you want to run FreeNAS with reliability.

You might want to consider looking into that first, although depending on how your RAID card is working with the disks, you may need to keep it like that to continue working with the disks (I assume you're in a no backup situation).

Read this:

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.ixsystems.com

Point 5 is pertinent to you and point 6 may explain what happened if you have any caching going on and the box was somehow not cleanly shut down and/or data was lost from the RAID card's cache while you did your maintenance.

The linked illumos error description mentions this:
------------
If this error is encountered during 'zpool import', and if no recovery option is mentioned, the pool is unrecoverable and cannot be imported. The pool must be restored from an appropriate backup source. If a recovery option is available, the output from 'zpool import' will look something like the following:

# zpool import share
cannot import 'share': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Sun Sep 27 12:31:07 2009
should correct the problem. Approximately 53 seconds of data
will have to be discarded, irreversibly. Recovery can be
attempted by executing 'zpool import -F share'. A scrub of the pool
is strongly recommended following a successful recovery.


Recovery actions are requested with the -F option to either 'zpool clear' or 'zpool import'. Recovery will result in some data loss, because it reverts the pool to an earlier state. A dry-run recovery check can be performed by adding the -n option, affirming if recovery is possible without actually reverting the pool to its earlier state.
------------


It seems to me you could run zpool import with the -F -n option to see if the pool can be recovered (possibly with data loss... but the n switch indicates that nothing will be changed, only checked to see if something can be done). Based on the overall statement, it looks like your chances aren't good.

Is there a better way of going forward on this?

Not without a time machine (an actual one, not the apple thing). It looks like you might be looking for a recovery tool... I think this article might help with that... https://www.ixsystems.com/community/threads/update-11-3-zfs-pool-empty.85299/post-590458

Should I stay on the config from two weeks ago or revert to the one I created once I found the fault?

Unless you changed settings after that config backup was taken, there will be no reason to attempt to change. The data (including metadata) in your pool is not related to that config.

 

[Dotball]

Thank you for the detailed response, I'm not enjoying the sound of this at the moment.

When I run
zpool import -Fn Vault
it returns a null response. As in it immediately returns to the command prompt with no output. Without the -n it says

Code:

root@pantry:~ # zpool import -F Vault
cannot import 'Vault': I/O error
        Destroy and re-create the pool from
        a backup source.
root@pantry:~ #

The interface at storage>pools show Vault as unknown and none of the drives are currently associated with any pool.

In terms of the time machine I have the plans but some of the intricacies are beyond my abilities.

 

[sretalla]

In terms of the time machine I have the plans but some of the intricacies are beyond my abilities.

I think the secret is in the flux capacitor.

 

[sretalla]

The interface at storage>pools show Vault as unknown and none of the drives are currently associated with any pool.

Depending on how far you want/need to go with it, you may want to use dd to take an image of those drives to another location (assuming you can throw together such a thing with sufficient capacity). You could then use those img files to feed into recovery software (like Klennet as linked in the post above... the free version lets you see if it will work and what can be recovered, then you need to paid version to get to your data)... you will need a Windows machine to use that in all likelihood.

 

[Dotball]

I'm fairly sure the raid controller is behaving as a HBA adaptor as I can get valid responses from camcontrol devlist as well as smartctl requests. That is the first test according to the link you provided.

I still have the boot environments from 2017 and various versions (9.10, 11.1, 11.3, etc.) to the current one. Can I revert to one of those?

Noticed one option in a previous version was a manual setup for a zfs array. Would that allow bypassing the drive initialization?

Is "dd" an internal command? Do you have any reference I could read please?

Thanks again

 

[jgreco]

I'm fairly sure the raid controller is behaving as a HBA adaptor as I can get valid responses from camcontrol devlist as well as smartctl requests. That is the first test according to the link you provided.

This isn't a pick-one-option situation.

For example, if we were to consider the pandemic, not having a nasty cough doesn't prove you don't have 'rona. The headaches and 101'F temperature that you do have are also signs, and the lack of a nasty cough wouldn't prevent you from being told to self-quarantine.

You're not the first person to optimistically decide that the first point overrode what came later so I guess I'll have to restructure the thing.

 

[Dotball]

Sorry, but it seemed there were a few sections which pointed to HBA, I'm happy to be corrected but it has been working to date so I had no real reason to doubt it wouldn't continue. If I had my time again it might be built differently but I am still waiting on the flux capacitor I have now ordered.

From my experience, in a hardware world you create an array using physical disks then you initialize the disks and finally you format them for your FS of choice.

What I am now trying to wrap my head around is this whole software raid system using ZFS. Can someone please tell me if I click create on the below screenshot does it initialize the disks or does it just create the dataset?

Then if I move to a CLI this command looks as though it would do what I am hoping.
zpool create Vault2 raidz /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5

Please do not take my approach the wrong way, I recognize the expertise here is way in advance of what I may, or may not, know but I do have a good range of experience in the computing world and I am calling on previous experience in a different environment.

My question also does still stand about loading a previous version to "refind" the pool. What I am trying to avoid at the moment is doing a destructive process which should have been avoided.

Thanks again

 

[Redcoat]

Is "dd" an internal command? Do you have any reference I could read please?

Google "freebsd dd" for the link to the man page.

 

[sretalla]

Can someone please tell me if I click create on the below screenshot does it initialize the disks or does it just create the dataset?

You should see a warning if you do that that all data on those disks will be wiped if you do that.

Then if I move to a CLI this command looks as though it would do what I am hoping.
zpool create Vault2 raidz /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5

You may not see a warning, but same outcome as above (although you will not see the pool in the GUI until you export it at the CLI and then import in the GUI if you go that way, again to stress, without your pool's data).

My question also does still stand about loading a previous version to "refind" the pool. What I am trying to avoid at the moment is doing a destructive process which should have been avoided.

You either need to use dd (like dd if=/dev/mfid0 of=/mnt/pool/location/mfid0-file.img bs=1024k ... will produce an .img file which is an exact block by block replica of the disk... vary the bs setting to equal the actual block size if you know it... but doesn't matter in the end in terms of the resulting file) or another tool to make an image of the disks before you take any action with them if you don't want to risk losing what may be your only copy of the data.

Using a tool (like Klennet) to recover the data could be done with the original disks themselves or with the .img files if you can make that work. It seems there will be no other path to getting your data back... old configs will not help you with that, it's not config, but rather parts of the metadata on the pool disks that's bad.

 

[jgreco]

Sorry, but it seemed there were a few sections which pointed to HBA, I'm happy to be corrected but it has been working to date so I had no real reason to doubt it wouldn't continue.

ZFS is *extremely* *demanding* on controllers. That stuff like patrol reads or RAID rebuilds that would be done by a hardware RAID controller is done instead by ZFS on the host platform, meaning that ZFS will often want to read dozens or hundreds of terabytes from all the disks. The real problem is that lots of RAID controllers are dodgy when you get to things like a disk I/O timeout or read error, or when a disk needs to be offlined and replaced. Some controllers don't support hotswap correctly, or maybe their FreeBSD drivers don't, or other "edge case" things go sideways.

This is why "it works - look, I built a pool and wrote some data" isn't the measuring stick we recommend you use. Cyberjock, when he was new, walked in here with his Areca RAID card, and admittedly it was a really nice card, with a vendor-supplied driver, but eventually discovered some relatively annoying problem that caused him to move to LSI.

Any disk controller has to work with SATA. For RAID/HBA, it also has to work with SAS and expanders. It can't stall when a disk experiences an I/O failure. It can't implement its own policies about retries. It needs to pass traffic back and forth to drives without getting unduly involved in the process. It can't put your data in an enclosing dedicated partition scheme. And the entire subsystem (card plus driver) needs to work as close to 100.0_% correctly on FreeBSD as possible.

The LSI cards crossflashed to IT mode with the exact driver/firmware we specify do this. Intel SATA PCH ports do this. Intel SCU PCH ports do this. Some (but not all) Marvell SATA controllers do this. AHCI SATA controllers have a good chance of doing this, depending on how true they are.

Lots of stuff seems to work, until it doesn't. We use a more stringent measuring stick because we assume people are storing valuable data that they do not want to lose.

People sometimes accuse us of being nasty narrow-minded snobby bastards about hardware compatibility but it isn't because we're trying to be jerks just for the sake of being jerks. It is because when things go off the rails, too often it gets to a state where there is significant damage or complete unrecoverability, and this is very distressing and disappointing to those of us who are trying to support FreeNAS users. I don't recommend LSI HBA's crossflashed to IT mode because I own LSI stock (I don't). I recommend them because they're the thing that's known to work correctly.

Very sorry this has happened to you, and I wish you good luck with your recovery.

 

[Dotball]

OK thank you, I have reached out to Alexey with some questions.

Looks like I need to buy some hardware and, obviously, Klennet.

Thank you for putting up with me. At least I learnt some new stuff in this. <thumbs up>

 

[Dotball]

ZFS is *extremely* *demanding* on controllers. That stuff like patrol reads or RAID rebuilds that would be done by a hardware RAID controller is done instead by ZFS on the host platform, meaning that ZFS will often want to read dozens or hundreds of terabytes from all the disks. The real problem is that lots of RAID controllers are dodgy when you get to things like a disk I/O timeout or read error, or when a disk needs to be offlined and replaced. Some controllers don't support hotswap correctly, or maybe their FreeBSD drivers don't, or other "edge case" things go sideways.

This is why "it works - look, I built a pool and wrote some data" isn't the measuring stick we recommend you use. Cyberjock, when he was new, walked in here with his Areca RAID card, and admittedly it was a really nice card, with a vendor-supplied driver, but eventually discovered some relatively annoying problem that caused him to move to LSI.

Any disk controller has to work with SATA. For RAID/HBA, it also has to work with SAS and expanders. It can't stall when a disk experiences an I/O failure. It can't implement its own policies about retries. It needs to pass traffic back and forth to drives without getting unduly involved in the process. It can't put your data in an enclosing dedicated partition scheme. And the entire subsystem (card plus driver) needs to work as close to 100.0_% correctly on FreeBSD as possible.

The LSI cards crossflashed to IT mode with the exact driver/firmware we specify do this. Intel SATA PCH ports do this. Intel SCU PCH ports do this. Some (but not all) Marvell SATA controllers do this. AHCI SATA controllers have a good chance of doing this, depending on how true they are.

Lots of stuff seems to work, until it doesn't. We use a more stringent measuring stick because we assume people are storing valuable data that they do not want to lose.

People sometimes accuse us of being nasty narrow-minded snobby bastards about hardware compatibility but it isn't because we're trying to be jerks just for the sake of being jerks. It is because when things go off the rails, too often it gets to a state where there is significant damage or complete unrecoverability, and this is very distressing and disappointing to those of us who are trying to support FreeNAS users. I don't recommend LSI HBA's crossflashed to IT mode because I own LSI stock (I don't). I recommend them because they're the thing that's known to work correctly.

Very sorry this has happened to you, and I wish you good luck with your recovery.

Thank you, I appreciate the time you took to pen that reply it made a lot of sense.

The situation when I started down this path was I had an old server (Dell R710) with all the "right" hardware for it at the time. I added the WD Red drives and built it.

Freenas has, clearly, evolved over time and the hardware has become more critical. My little server continued to run unbeknownst to me it was not "compatible".

I understand you need to make a line call on what can, and what cannot, be "supported hardware" and I do not have an issue with that. Nor do I find it "snobby", it is just necessary and I agree with you.

Whether my hardware needs to be replaced or not it is my fault I am in this situation as I failed the first rule of IT, make sure you have valid backups. Once I recover what I can from this I will look at options on replacing the controller, etc. and take that all you have detailed above on board.

As I said to sretalla, at least I have learnt things from this. <thumbs up>

Thanks again.