SOLVED Zpools online, kernel panics when I try to zpool import

[Mehul V]

Hi all,

I needed some advise on what the best solution would be for me to move some or all of my data out of my now corrupted FreeNAS so that I can wipe/format the OS and data drives and build a new clean system.

Here is the hardware we have available to move data to/from:

1. FreeNAS-main -- 40TB RAW (using RaidZ2 it's about 94% full) - FreeNAS (FreeNAS-11.1-U4) installed
2. FreeNAS-sandbox01 -- 32TB RAW empty - FreeNAS (FreeNAS-11.1-U4) installed
3. FreeNAS-sandbox02 -- 32TB RAW empty - No OS installed

Current status on FreeNAS-main system:

• After multiple hard resets and random shutdowns, the system is now unable to boot up to the main page with 11 options and a webURL to access to GUI.
• Bootup is halted at some point where I am left with db> prompt.

Here is what I have tried so far:

• I tried continue and reset options at the db> prompt but it didn't help.
  
• Then I tried to boot up the installer USB so that I can select shell from it. Was stuck on "GRUB" screen but was able to resolve the issue with the help of THIS thread
• Once loaded, I selected #3 "SHELL" and according to THIS thread, I typed in #zpool import and was able to see my raidz2 pool called media with all the drives ONLINE

Here is what I need help with:

• I would like to import this pool then copy the data onto my new 80TB drives that I have yet to install
• Is it possible to take my 40TB drives (10 x 4TB) and move the HDDs from FreeNAS-main to FreeNAS-sandbox01 and get it up and running on FreeNAS-sandbox01?
• This way, I would have FreeNAS-sandbox01 with my old data and I can format/wipe the OS on FreeNAS-main, install (10 x 8TB) and copy/rearrange the data back from FreeNAS-sandbox01 to FreeNAS-main

I tried my best to be as detailed as possible here, but if I missed something or if anyone needs more info, please let me know and I can get that right away.
Any help would be appreciated.

Thanks,
Mehul

 

[Jailer]

Your data should be fine as you pool is separate from the OS. Save your config and reinstall FreeNAS to your boot device and upload your saved config and you should be back in business.

 

[Mehul V]

Your data should be fine as you pool is separate from the OS. Save your config and reinstall FreeNAS to your boot device and upload your saved config and you should be back in business.

Unfortunately I am not able to access to GUI to backup the latest config file. Is there a different way to back it up?

 

[Jailer]

It is backed up if I'm not mistaken and should be located in your pool at /data/freenas-v1.db

 

[danb35]

It is backed up if I'm not mistaken and should be located in your pool at /data/freenas-v1.db

freenas-v1.db is the live copy on your boot device (which is also a pool, of course). The backups are at /var/db/system/configs-longhexnumber/.

 

[Mehul V]

I just tried to import the pool using #zpool import -f media and I got stuck at db > prompt again.
Attached is a screenshot of the error.

 

[Jailer]

Reinstall the same version of FreeNAS to another boot device and import your pool so you can recover your config from there.

 

[Mehul V]

Reinstall the same version of FreeNAS to another boot device and import your pool so you can recover your config from there.

Installed same FreeNAS on another device and I was able to get to the UI. Logged into the web interface and there were no volumes under storage tab. I click Import volume and selected my volume from dropdown. As soon as I hit next, the UI said "Some error occured" and the console got stuck at attached screenshot.
The system then rebooted and I am back at the db > prompt on the console.

 

[Mehul V]

I moved all 10 HDDs to a different system, installed clean FreeNAS on a separate boot SSD and when I boot into it, the system can see the 10 HDDs during boot up but using the "View Disks" options in the FreeNAS GUI does not see any of the 10 HDDs. What am I doing wrong here? Instead of progressing, it seems I am going in reverse :(


Update: Changed the raid controller to JBOD and disks are now showing. The clean install works fine until I try to import the volume, then kernel panic happens and I am back to db > prompt.
Last thing I tried was zpool import -fFn -R /mnt media and panic happened and I am back at db > prompt.

It has now been 14 hours since I started troubleshooting and I have gotten much progress. I am starting Memtest86 and going to bed. Hopefully someone here can provide some advise on what I should try next.

-Mehul

 

[Jailer]

Changed the raid controller to JBOD

Well this would have been good to know from the start and is one of the reasons the forum rules state to post complete hardware specs when asking for help. If this pool was created using a RAID card you're pretty much out of luck. FreeNAS requires direct access to the disks and a RAID card does not provide that.

 

[Mehul V]

Well this would have been good to know from the start and is one of the reasons the forum rules state to post complete hardware specs when asking for help. If this pool was created using a RAID card you're pretty much out of luck. FreeNAS requires direct access to the disks and a RAID card does not provide that.

I apologize, I should have been more clear about what I did. The main system, we're calling it FreeNAS-main originally had the 4TB x 10 in it. In order to rule out any bad hardware or bad boot device, I physically removed the drives from FreeNAS-main and installed them into FreeNAS-sandbox01. This new machine is the one that I had to change the controller to JBOD to make it work similarly as the main FreeNAS machine.
Theoretically, I should be able to install FreeNAS on FreeNAS-sandbox01 and just import the volume from the old drives, right?

 

[garm]

What is your system specs?

 

[Mehul V]

What is your system specs?

Specs for FreeNAS-main:

• 24 Bay FreeNAS Chenbro RM41824 4u Server
  
• 2 x Intel Xeon E5-2650 2.0GHz 8 Core CPU
• 3 x LSI SAS9211-8i Raid Cards
• 1 x 1200W Power Supply
• 8 x 8GB Ram DDR3 ECC + 4 x 16GB DDR3 ECC

Specs for FreeNAS-sandbox01:

• Ci Design NSR316 chassis (3RU rackmount)
  
• Supermicro X8DTHiF motherboard
  
• 2 x Intel Xeon E5520 2.26GHz 4 Core CPU
  
• 6 x Hynix 8GB DIMMs (1066) (48GB total)
  
• 3Ware 9650SE16ML SATA RAID controller (currently in JBOD) w/ battery backup option installed
  
• SATA DVD drive
  
• OCZ ZDrive PCIe SSD

Here is a video link that shows the pool is online but I get kernel panic when I try to import it as readonly - https://www.youtube.com/watch?v=UTaSOrA4aq0

 

[garm]

What’s the firmware of the LSI cards?

 

[Mehul V]

What’s the firmware of the LSI cards?

Not sure, but I will check and find out. Currently FreeNAS-main is turned off and I am working on FreeNAS-sandbox01 with the 10 drives installed. I would like to get this working on FreeNAS-sandbox01 so that I can build a new system on FreeNAS-main and migrate the data over there.


I installed live version of mfsbsd on USB and booted into it.
zpool import still shows the "media" pool online and all 10 HDDs available, but when I try to import and mount it on /mnt/media, kernel panics and system reboots.

While going over THIS thread, I ran into a helpful command that shows what might be causing the kernel panic.

After running zdb -e -F media, I get the following:

Code:

error: blkptr at 0x837f42d00 has invalid CHECKSUM 0
pid 39 9=(zdb), uid 0: exited on signal 6 (core dumped)
Abort trap (core dumped)

Attached is the screenshot of the error.

 

[Mehul V]

What’s the firmware of the LSI cards?

The FW Revision on all three SAS9211-8i LSI cards is: 14.00.00.00 - IR

 

[Mehul V]

Updated title to reflect the issue now that we know a little more about the situation.

 

[garm]

Can’t help you, you are running an old and unsupported firmware for those LSI cards. To be clear, you haven’t given ZFS full access to the drives. You are in a very bad spot and anything I say would jeopardize data. My two cents are to find a data recovery firm specializing in ZFS. They might be able to help you rebuild the array you based the pool on.

 

[Mehul V]

Can’t help you, you are running an old and unsupported firmware for those LSI cards. To be clear, you haven’t given ZFS full access to the drives. You are in a very bad spot and anything I say would jeopardize data. My two cents are to find a data recovery firm specializing in ZFS. They might be able to help you rebuild the array you based the pool on.

Is this because the raid controller is in IR mode? Can I install the drives back in it's original system, change the controller to be in IT mode and then try to import the pool?

Going to a data recovery firm would be too expensive and is not an option for me.

 

[Jailer]

It depends on how your pool was created. But with it panicking at this point i dont have much confidence that it will work.

Having said that it would be good to get the opinion of some more knowledgable members such as @Ericloewe or @Stux or @Spearfoot and see what they think.