Pool lost disks after crash of power supply

[c.fuhrmann]

Ok - first I give the report on "dmesg": See attached file.

So I should do: "zpool import -f -o readonly=on" in one?


Building a new FreeNAS with save optinons is a thing after saving the data...

 

[HoneyBadger]

The dmesg makes it very clear that you were using virtual disks on local datastores. Like so many things technological, it works great, until it suddenly doesn't. The "onboard SATA controller/local HDD" combination is virtually guaranteed to use drive write cache, and with spinning disks not being power-fail safe, a sudden power loss in the middle of write activity is expected to cause corruption.

Try importing your pool as read-only with the -f (force) option set. If this does not work you may have to get more aggressive with import options such as rollback and selecting older uberblocks (which discards all data newer than that timestamp) - this is assuming that your metadata is still salvageable at that transaction group.

 

[Scharbag]

"You are using ESXI. How are you passing the disks into the VM? This should be done with some sort of hardware passthrough that needs VT-d. "
I hope this picture shows what you need:
View attachment 42847
The other HDDs are similar "plugged" into the FreeNAS-machine.

The storage in ESXi for this HDD is:

View attachment 42848
This system worked very well the last years...


"Also, could you post the output of zpool status?"
I hope this is what you need:
View attachment 42851

Chris Fuhrmann

Derp...

As others have said, you got lucky for a while...

To run FreeNAS successfully on top of ESXi, you MUST pass through physical drive hardware to the FreeNAS VM. This is best done by using an HBA like an LSI 9211-8i flashed to IT mode. If that is done properly, FreeNAS has full access to the storage just like it would on a bare metal installation.

Here is a good guide on how to set up FreeNAS on ESXi. It is a bit dated but it is accurate.

Good luck on getting the system to import the volume. Unfortunately, this is just one more opportunity to say that RAIDZn is not a replacement for a backup.

Cheers,

 

[c.fuhrmann]

Ok - after all I got help via TeamViewer from Michael Dexter, Gainframe.
I'm not shure to repeat all steps, but:
He unmounted the disks from FreeNAS and then from ESXi. Then investigating all disks / volumes. Fopund no errors. Mounting them agin and adding in FreeNAS. Then "zpool import" - this showed (after all unmounting etc.) there are data avaliable from DATE - and the date was exactely the time, when the NAS power supply crashed. Imported it and all was good!

I'm now backing up all my data twice. Afterward I'l build a new system. First installing a bare FreeNAS on my hardware and on a stick to start the NAS without any other HDD. Second installing (the newest) ESXi and on that installing FreeNAS with passthrough for the HDDs. Thats just the better way, everyone tells!

 

[c.fuhrmann]

Thanks everyone for the hepl here - I've learned a lot!

 

[Scharbag]

Ok - after all I got help via TeamViewer from Michael Dexter, Gainframe.
I'm not shure to repeat all steps, but:
He unmounted the disks from FreeNAS and then from ESXi. Then investigating all disks / volumes. Fopund no errors. Mounting them agin and adding in FreeNAS. Then "zpool import" - this showed (after all unmounting etc.) there are data avaliable from DATE - and the date was exactely the time, when the NAS power supply crashed. Imported it and all was good!

I'm now backing up all my data twice. Afterward I'l build a new system. First installing a bare FreeNAS on my hardware and on a stick to start the NAS without any other HDD. Second installing (the newest) ESXi and on that installing FreeNAS with passthrough for the HDDs. Thats just the better way, everyone tells!

Awesome to hear that you were able to recover your system.

Moving forward, the combination of ESXi and HBA passthrough is a very viable option. I have been running this in production (personal server) for many years now. I boot ESXi from a USB key. I have 2 SSDs that are used for ESXi local datastores and this is where I store the FreeNAS VM as well as my pfSense VM. So far it has been very reliable (touching wood) and easy to maintain. Take a look at my signature for the layout.

Awesome part is that once FreeNAS boots, all of my other VMs become available via NFS shares from FreeNAS back into ESXi. I chose NFS for simplicity. iSCSI is another option but for my needs, it does not add significant value.

Anyway, happy FreeNAS-ing!

 

[Andrew Corser]

Awesome to hear that you were able to recover your system.

Moving forward, the combination of ESXi and HBA passthrough is a very viable option. I have been running this in production (personal server) for many years now. I boot ESXi from a USB key. I have 2 SSDs that are used for ESXi local datastores and this is where I store the FreeNAS VM as well as my pfSense VM. So far it has been very reliable (touching wood) and easy to maintain. Take a look at my signature for the layout.

Awesome part is that once FreeNAS boots, all of my other VMs become available via NFS shares from FreeNAS back into ESXi. I chose NFS for simplicity. iSCSI is another option but for my needs, it does not add significant value.

Anyway, happy FreeNAS-ing!

No, wait...sorry for this completely newbie question, but did I gather from this discussion that if I use FreeNAS, and someone inadvertantly switches off the NAS at the plug, that the whole things falls over when you try to reboot it? If so, then perhaps FreeNAS isn't a local replacement for my use of (surveillance capitalist) Google Drive..?

 

[HoneyBadger]

No, wait...sorry for this completely newbie question, but did I gather from this discussion that if I use FreeNAS, and someone inadvertantly switches off the NAS at the plug, that the whole things falls over when you try to reboot it?

Welcome,

In this case, the OP had inadvertently built himself a metaphorical "house of cards" by installing FreeNAS as a virtual machine on top of ESXi. The instability and the necessity to do some shenanigans with respect to the "unpresent drive, reattach drive, import" is entirely a result of the use of VMware virtual drives, which is strongly discouraged for stable environments.

A bare-metal FreeNAS setup, or a FreeNAS VM configured with an HBA in passthrough mode, is far more resilient to unexpected power loss. Obviously if you are in the process of copying a file to the system, and immediately kick the power out when it finishes, you can put yourself in a situation of that file being corrupted (assuming the default of asynchronous writes is used) but existing files will be fine. More resilient systems can be built using things like log devices, ensuring that any data written to the system is kept in a stable write log.

 

[Scharbag]

No, wait...sorry for this completely newbie question, but did I gather from this discussion that if I use FreeNAS, and someone inadvertantly switches off the NAS at the plug, that the whole things falls over when you try to reboot it? If so, then perhaps FreeNAS isn't a local replacement for my use of (surveillance capitalist) Google Drive..?

As @HoneyBadger states, the OP set himself up for failure by not reading the plethora of posts that clearly tell people to NEVER EVER do what he did in a production environment!!

Personally, I have been running FreeNAS (TrueNAS now) on an ESXi (5.5, 6.0, 6.5) system with HBA passthrough for many years. Due to a faulty UPS, I have had the power suddenly fail to my ESXi system and, touch wood, nothing has ever gone wrong. I host multiple VMs on TrueNAS storage and they are stable, even after unexpected power failures.

All in all, TrueNAS has been a fantastic system for me. There are some pitfalls that can get you. Some are just annoying, some are disasters waiting to happen. That said, these forums are full of great advice and helpful people. Do a bit of reading and research before you decide to take the TrueNAS and/or ZFS plunge. One downside of ZFS is it is unforgiving. Like a vicious kick in the nuts unforgiving. Even now, I wish I had setup my production pool as a pair of 8 disk Z2 vDevs instead of 6 disks (I backup everything to a pool with 2@9 disk Z2 vDevs so I could push the limits on my production pool...). For me to adjust this now is a metric F-tonne of time and effort... But, such is life. Perhaps some day the ZFS magicians will figure out how to add a drive to a vDev (I will hold my breath...).

Cheers,

 

[Andrew Corser]

As @HoneyBadger states, the OP set himself up for failure by not reading the plethora of posts that clearly tell people to NEVER EVER do what he did in a production environment!!

Personally, I have been running FreeNAS (TrueNAS now) on an ESXi (5.5, 6.0, 6.5) system with HBA passthrough for many years. Due to a faulty UPS, I have had the power suddenly fail to my ESXi system and, touch wood, nothing has ever gone wrong. I host multiple VMs on TrueNAS storage and they are stable, even after unexpected power failures.

All in all, TrueNAS has been a fantastic system for me. There are some pitfalls that can get you. Some are just annoying, some are disasters waiting to happen. That said, these forums are full of great advice and helpful people. Do a bit of reading and research before you decide to take the TrueNAS and/or ZFS plunge. One downside of ZFS is it is unforgiving. Like a vicious kick in the nuts unforgiving. Even now, I wish I had setup my production pool as a pair of 8 disk Z2 vDevs instead of 6 disks (I backup everything to a pool with 2@9 disk Z2 vDevs so I could push the limits on my production pool...). For me to adjust this now is a metric F-tonne of time and effort... But, such is life. Perhaps some day the ZFS magicians will figure out how to add a drive to a vDev (I will hold my breath...).

Cheers,

Many thanks for the helpful replies (Scharbag and HoneyBadger)! I will take my time to go through the details, and see if TrueNAS Core is what I need...I suppose the main question for me is whether my requirements are so minimal compared to the capability of TrueNAS, which is clearly able to cope with enterprises and large storage, that I should be looking for something simpler. I just thought I would like a home media server, and home cloud and backup (to get out from under Google Drive's synch and backup - I have escaped Gmail by moving to ProtonMail) making use of a couple of old laptops/pc's and possibly some external storage bits I have...but then the old laptop I was using powered down, and the initial pool I had set up (using usb sticks) was lost. No big deal, but possibly indicative of looking at the wrong solution to my little question!

 

[Andrew Corser]

@Scharbag @HoneyBadger ... thanks again (see above). Would you say that TrueNAS Core is a sledgehammer to crack a nut in my case - backed-up network storage of no more than 1 TB and about the same amount of storage again for a media server; no need for differential permissions for a maximum of 5 users?

 

[Scharbag]

@Scharbag @HoneyBadger ... thanks again (see above). Would you say that TrueNAS Core is a sledgehammer to crack a nut in my case - backed-up network storage of no more than 1 TB and about the same amount of storage again for a media server; no need for differential permissions for a maximum of 5 users?

ZFS may be overkill for what you are looking for... That said, TrueNAS does a lot more than just store files. It is pretty cool for a lot of things.

Cheers,

 

[Heracles]

@Scharbag @HoneyBadger ... thanks again (see above). Would you say that TrueNAS Core is a sledgehammer to crack a nut in my case - backed-up network storage of no more than 1 TB and about the same amount of storage again for a media server; no need for differential permissions for a maximum of 5 users?

Hey @Andrew Corser,

What is important for you to note here is the need for backups. No single server, FreeNAS / TrueNAS or other, can be more than a single point of failure. When you use a commercial cloud like Google, they protect themselves against physical incidents like a fire in a datacenter. They also have very rigourous procedures and risk of logical incident is low.

Should you run your own data storage service, you will not have such a protection unless you deploy yourself a complete backup strategy (see my signature for that).

Also, in this very case, such a complete backup would also have protected the OP here. Should he had a complete backup, he could just have restore his data from it.

Backups, backups, backups.... For so many things, a simple if not the only solution is to use your backups. Also, know that a backup is not working until you successfully restored it at least once.

Never think that a single server can relieve you from the need of doing your backups.

 

[jgreco]

When you use a commercial cloud like Google, they protect themselves against physical incidents like a fire in a datacenter. They also have very rigourous procedures and risk of logical incident is low.

While that's generally true, there's no particular guarantee that your data is safe, secured, or protected from harm. The cloud is subject to failure; after all, it is just putting your stuff on someone else's computers and hoping that they're better than you, but the cloud will cry few tears at the loss of your bits.

 

[Heracles]

While that's generally true, there's no particular guarantee that your data is safe, secured, or protected from harm.

Indeed. Even more when you look at their EULA where they claim the right to modify your data, publish them and more. That's why I designed, deployed and use my private cloud.

But I allow myself to rely on it completely with all my data only because I designed a complete and professional grade solution that I test regularly, including the backup and restore plan.

Without such a complete strategy, the residual risk associated with professional clouds is lower than the residual risk of a single point of failure designed, built and operated by a single guy.

 

[Scharbag]

I use Tresorit to backup my irreplaceable data to the cloud. It is supposed to be client side encrypted so there is no way for the server side to access data. Also, Tresorit is hosted in Ireland and the Netherlands.

Locally, I backup all of my replaceable data on my server to a backup pool. Yes, if I have a fire/flood/theft and loose my server, I will have a lot of work to do. But I will not loose anything that I cannot get back with time and effort.

Cheers,