KDB Panic on pool import

[turboaaa]

Running a FreeNAS MIni-XL with 8x4TB Toshiba drives in a raidz2 pool (This is for archival purposes) and the official FreeNAS ZIL SSD.

I was in the middle of a zfs send over ssh, and a robocopy job over smb, to the mini-xl when suddenly the system crashed and rebooted. After the reboot I started the send again. While it was running I looked at dmesg and found ATA errors, the web ui reported unreadable sectors on one of the drives. Before I could do anything else the system crashed, but this time would panic when importing the pool. I have never seen this before and am a bit purplexed.

I tried removing all the disks, booting to the OS, then importing using the read only flag. Sadly it tells me I must destroy the pool and restore from backups. Any hope or am I to start from scratch?

EDIT : The pool is encrypted, I have the keys backed up.

FREENAS-MINI-XL
OS Version:
FreeNAS-11.2-U6
(Build Date: Sep 17, 2019 0:16)
Processor:
Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (8 cores)
Memory:
32 GiB

 

[turboaaa]

I think the read only import failed because I didn't have the disks decrypted. Using the GUI to decrypt caused the system to crash again. I will look up how to decrypt using the cli.

 

[PhiloEpisteme]

The command you're looking for is geli attach -k /path/to/key /dev/gptid/{device}. If you're using a key without a passphrase add the -p flag.

 

[turboaaa]

The command you're looking for is geli attach -k /path/to/key /dev/gptid/{device}. If you're using a key without a passphrase add the -p flag.

lol, I JUST figured that out and was about to report on it. Thank you for the quick reply regardless.

Yes, I was able to decrypt the disks using the above command. I was then able to import as read only. Now the fun part is to decide the best way to get my data without risking the same thing happening to my other freenas server. The other one is much beefier but this makes me VERY nervous.

Thankfully the data is not critical, but I would have to start replicating files over the VPN which would take another month to get caught up :-(

 

[turboaaa]

I just imported the pool without the read only flag, other than using the default mount point (/<poolname>) it appears to be functioning. When I decrypted I used /dev/adaxpx instead of gptid. The indicator lights on all the drives are blinking so I will wait for it to calm down before exporting and starting over with the gptid.

Hopefully after that step I will be able to reboot without a crash.

 

[turboaaa]

I exported the pool, shutdown the server, pulled the drives, booted back into freenas, inserted the drives, decrypted using gptid (I don't think this is necessary at this time, but I like consistency when stuff breaks without known reason), and imported the pool specifying an alternate mount root.

Since the first panic I had removed the drive with read errors, and it is still removed. A replacement should arrive Monday while I send this one back to Toshiba.

After the scrub I will attempt to reboot to see if the panic persists. I assume it will since I have not made any changes. Any ideas as to why the panic at boot during pool import, a panic using the web ui to import, and no panic when using the cli?

 

[PhiloEpisteme]

After the scrub I will attempt to reboot to see if the panic persists. I assume it will since I have not made any changes. Any ideas as to why the panic at boot during pool import, a panic using the web ui to import, and no panic when using the cli?

I don't know why you would get a panic, what was the exact error message and exactly when in the process did it start?

Also, I feel the need to point out that when you replace that drive it will require that reset the passphrase AND regenerate the recovery key so that the new drive has the same keys as the others.

To clarify, did you do the import via the GUI? If you did so while the drives were unlocked it is possible that FreeNAS doesn't know these are encrypted.

 

[turboaaa]

I don't know why you would get a panic, what was the exact error message and exactly when in the process did it start?

"KDB: Enter: Panic". Line right before it says something about a trampoline, and the lines before that appear to have something to do with condensing/compressing data. If it happens again during my test reboot this morning I will take a picture.

Also, I feel the need to point out that when you replace that drive it will require that reset the passphrase AND regenerate the recovery key so that the new drive has the same keys as the others.

Thank you for making sure this is known. Yes, I have many copies of backup keys for every time I have replaced a drive. I wasn't sure if it was necessary to backup the recovery key every time I replace a drive, good to know I was doing it right.

To clarify, did you do the import via the GUI? If you did so while the drives were unlocked it is possible that FreeNAS doesn't know these are encrypted.

Yes, I did attempt to import using the GUI. If I boot the system with the drives removed, then re-inserting the drives after boot, the pool shows up in the GUI as locked. Using the recovery key I was able to unlock the pool and begin importing. This would result in another crash and the system would reboot.

I agree that FreeNAS is the issue as geli, BSD, and zfs work on their own when manually going through the steps. My config is not too complex and can be relatively easily replicated. If it continues to crash I will perform a fresh install of FreeNAS, import the pool using the recovery key, and recreate my SMB/NFS config. However, I would love to know what is going on and how to fix it. If I get the error message, log files, or anything else that could give you insight let me know.

Thanks.

 

[turboaaa]

Rebooted and no panic. Started the same zfs receive that was running last time this happened. Will report results when finished, if it finishes it won't be until tomorrow as it is sending 1.5T over 1Gbps.

 

[turboaaa]

lol, I forgot what time of day it was when I posted this morning. The received finished with no problem.

At this point I am going to assume all is well, and that for some reason the drive with read errors caused this somehow. Still spooked but at least I can carry on. Have been using the thing for years and only had an issue after swapping out 2TB drives for 4TB ones. So going to blame the new drives.

 

[PhiloEpisteme]

Ah, I meant to reply last night, but it looks like all is well! Report back if you have other issues.