FreeNAS no longer responds when breaking the pool

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
Hey,

I'm currently playing with FreeNAS in a VM to see if I want to move my Storage Spaces pool over to it. My VM is running on Hyper-V with 8GB of ram. (I'm well aware not to run FreeNAS in a VM for production, this is just for testing only)

I'm trying to test different scenerios to see how well FreeNAS deals with them, and for me to know how to fix them, creating degraded volumes, resilvering etc.

So I created a Mirrored Striped Pool with 2 30GB HDDs (which are fixed sized virtual hdds stored on a SSD on the physical machine), created shares etc, all good.
I created another VDEV of 2 Mirrored Striped Pool of 2 40GB HDDs (again fixed sized virtual hdds stored on an SSD) and added them to the pool, all good.
Then I started to disconnect drives from the VM, to simulate drive failure. Removing one drive from each mirror worked fine, the volume showed as degraded and I was able to add a new drive and resilver. All good. But then I wanted to see what happened after a catastrophic pool failure, so I removed 2 HDDs from one VDEV, a complete failure state.

I expected FreeNAS to say the pool is dead. But what happened was FreeNAS essentially seems to crash. The GUI is no longer working, instead I get a message that
"Connecting to FreeNAS ... Make sure the FreeNAS system is powered on and connected to the network."
and eventually it even stops giving that message, just hanging on the logo.

If I connect to the console, I can type commands but they dont execute, the server feels like it's essentially dead, so I can't even cleanly reboot.

This doesn't seem to be how this should work, and it's a little worrying - I know obviously in this scenario the data is lost, but having the whole NAS crash is concerning, I could potentially have other pools on the NAS which this could corrupt potentially.

How do I fix this? Is this a bug? I don't like having to force reboot it, the first time this happened (I recreated it to see if it happened again, it did), it completely broke my Active Directory integration after the force reboot, and the only way I could fix it was to reinstall the OS, which isn't ideal.

Thanks!
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
No, this isn't a bug, but if you broke the pool containing the system dataset, then the middleware database that FreeNAS relies on will have disappeared while FreeNAS was operating, which isn't something it's designed to handle.

The only recovery from this is to reinstall, and to point the system dataset to a more robust pool.
 

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
No, this isn't a bug, but if you broke the pool containing the system dataset, then the middleware database that FreeNAS relies on will have disappeared while FreeNAS was operating, which isn't something it's designed to handle.

The only recovery from this is to reinstall, and to point the system dataset to a more robust pool.

Okay, of course I hope this never happens, but I'm just thinking about the worse case scenario. How do I safely shutdown the OS in this situation in order to protect other potential pools that are running? I assume a hard reset isn't good?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Okay, of course I hope this never happens, but I'm just thinking about the worse case scenario. How do I safely shutdown the OS in this situation in order to protect other potential pools that are running? I assume a hard reset isn't good?

ZFS can handle a hard reset; it'll replay the ZIL write journal on reboot to restore the file system to a consistent state.
 

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
Hey, thanks for the reply - if you don't have an SLOG drive, where is the ZIL journal stored? On the pool?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Yes, on the pool.
 
Top