Pool lost disks after crash of power supply

[c.fuhrmann]

Hi,
I got a pool "NAS" with 4 disks (da1, da2, da3, da4).
Two days ago I got the message:
New alerts:
* Pool NAS state is UNKNOWN:
Current alerts:
* Update Train EOL Reached
* Pool NAS state is UNKNOWN:

Some hours later the NAS was gone. Inspection showed: Power supply is off. And could not be started again.
So I installed a new power supply.

After starting I saw that my pool "NAS" has no disks no more.

In "Storage -> Disks" I can see all my disks. The formerly disk-allocation shows now: Name: da1, Pool: Unused (!), etc. (da2 to da4)

How do I "connect" the disks back to the pool without distroying the data on it?

Happy about help
Chris

 

[jgreco]

What size power supply are you using for this?

 

[c.fuhrmann]

Now a standard PC power supply 700W, 340W output at 12V.
How does the size of power supply depend on how to connect a "los" disk into a pool?

 

[jgreco]

It's not uncommon for people to be using undersized PSU's, which can cause pools to fail, which is why I wrote the Proper Power Supply Sizing Guide.

Part of getting your system back up and running is to make sure you're not setting yourself up to fail again. You can actually do permanent damage to your disks (rendering the data permanently gone).

 

[c.fuhrmann]

Ok - the system has run for more than 3 years without problems. And the former system very similar did it for more than 5 years. I think, the original power supply was at the end of life.
And reading your Guide I think a 340W at 12V supply is ok for 4 WD-HDDs and one SSD. How do you think?

And if you agree with my power supply - than the main question is: Ho do I "connect" the drives to their pool again?

 

[JaimieV]

Can you detail your system please?
The error "* Update Train EOL Reached" suggests you're on a very old version of FreeNAS.

Is your PSU three or eight years old? Is it a solid name brand or random "700W" no-name? If it's a decent one, 700W is certainly plenty - you're likely to be using less than 100W.

 

[c.fuhrmann]

Yes, I got a very old system of FreeNAS - sorry! I didn't find time for updating last months...

I got a
FreeNAS 11-2-U8 running on

• Intel Xeon E3-1220v6 (4*3,00 GHz) So 1151 with / on the main board
• Z!MB ASRock C236 WSI and
• 16 GB Hynix HMA82GU6AFR8N DDR4 RAM

all hardware very old from Feb 2019. After buying this in Feb. 19 I installed everything new, last updates Feb. 2020,
as a VM-machine
on an ESXi 6.7.0.

FreeNAS is running and reporting now, that "NAS is offline, not running scrub ".
So - my essential problem is, who to get the drives (da1, etc.) into the pool "NAS" back again without loosing the data.

Should I update to FreeNAS 11.3? Makes this the operation to include the drives into the pool again easier? I don't trust to do so myself. I thought it is better to let the system so, as it was, when it "lost" the disks....

 

[HoneyBadger]

What was your pool configuration beforehand (mirror, RAIDZ1/Z2)?

as a VM-machine
on an ESXi 6.7.0.

What method were you using to assign the disks to the FreeNAS VM?

 

[c.fuhrmann]

Oh shit - I knew that this question will come.
I'm not shure, what I did. I'm shure, that the data are double saved. But is this "mirror" or "RAID"? Where do I find this information?

When installing the system I think I entered "Import disk(s)" and added the 4 disks (2*1,8 TB, 2*3,8TB) to the pool.

 

[Bikerchris]

I have less than 1% of the knowledge of many on here, but perhaps you have 2 x mirrors (a 1.8TB mirror and 3.8TB mirror). You should wait until someone with more experience makes a suggestion, but I would try to import one pair of same sized disks and see what happens.

 

[c.fuhrmann]

When I try to import the disks into the pool (Storage - Import disk), than I get the options for the file system of the disk: UFS, NTFS, MSDOSFS, EXT2FS.
The disks are in ZFS.
So this seems to me not to be the right way....

And: Yes, I think also I got "Mirror". But I'm not really shure about...

 

[Bikerchris]

Have you tried unplugging all drives except for one and trying an import? If you're lucky, it might even be a cable problem.

 

[c.fuhrmann]

Yes - importing was the first thing, I tried. But as written up here,
when I try to import the disks into the pool (Storage - Import disk), than I get the options for the file system of the disk: UFS, NTFS, MSDOSFS, EXT2FS.
The disks are in ZFS.

 

[Scharbag]

You are using ESXI. How are you passing the disks into the VM? This should be done with some sort of hardware passthrough that needs VT-d. Please let us know.

Also, could you post the output of zpool status?

Cheers,

 

[Bikerchris]

Yes - importing was the first thing, I tried. But as written up here,
when I try to import the disks into the pool (Storage - Import disk), than I get the options for the file system of the disk: UFS, NTFS, MSDOSFS, EXT2FS.
The disks are in ZFS.

Ah, only just noticed the VM mounting, I would certainly suggest responding to @Scharbag comment.

 

[c.fuhrmann]

"You are using ESXI. How are you passing the disks into the VM? This should be done with some sort of hardware passthrough that needs VT-d. "
I hope this picture shows what you need:

The other HDDs are similar "plugged" into the FreeNAS-machine.

The storage in ESXi for this HDD is:

This system worked very well the last years...


"Also, could you post the output of zpool status?"
I hope this is what you need:

Chris Fuhrmann

 

[sretalla]

Maybe it has worked well for you (and you were riding the razor's edge with a lot of luck so far in that case), but it's now time to reconsider.

You're running virtualized, but you're using virtual disks from the VMware system to pass into FreeNAS... this is a recipe for disaster (and disaster may have already arrived for you).

I suggest that you first read this article:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Then I would suggest that you look at the output from zpool import to see if the disks are still in play.

Also look at dmesg to see if anything useful is being reported.

Importing a disk is not what you need in this case, it's importing a volume (I think that's how it was labeled in that version, even though it's actually a pool that you're importing)

If you manage to re-import your pool I wouldn't let it sit like it is any longer if you care about the data that you have stored there. You need to do a re-think according to the thread I referred above and pass the disk controller throguh to the VM, not individual virtual disks.

It does look like you have a pair of mirrors, probably one of 1.8TB and one of 3.6TB. They may be VDEVs in one pool, or two separate pools. We'll see if zpool import shows you anything.

 

[c.fuhrmann]

Thanks for your article. But before changing my system I need a backup of the data. Afterward....
So - now I'm hoping, that the disaster is not coming until I saved my data.

zpool import shows:

How do I get an output page by page? When I try dmesg, I get only the last part to read:

And: By all: What does this say to me?

The only thing I know: I have had just one (1) pool, named NAS....

Happy about your help!
Chris

 

[HoneyBadger]

If your networking is functioning, either enable the SSH service (and allow login with root+password) and connect via a terminal client (PuTTy is popular under Windows) or pipe the output of dmesg/other commands through less, as shown below

dmesg | less

Use Q to quit

 

[HoneyBadger]

As mentioned before, you've built this pool in a perilous manner (VMDKs on top of a VMFS) - the sudden power outage may have introduced corruption in your VMDKs.

You could attempt to import with zpool import -f but I would suggest also adding -o readonly=on to ensure that no further damage is done; assuming what has already happened can be recovered from.