FreeNas 9.10 ZFS pool won't import volume. panic: solaris assert

[Slydogsz]

Hello All,

Looking for some help as my FreeNas decided that I should spend my weekend trying to recover all of the data on it. I am running FreeNas 9.10 and during the boot when it starts to try and mount the pool it panics and throws me to the db> prompt.

If I run gpart list it shows all of my drives (4 3tb drives + 128gb ssd cache drive) as having a state of OK. I would post the gpart output here but I dont have anyway to capture it as I am working from the console. Smartctl on all of my drives returns a passed test result as well.

If i try to import the volume with a -f i get dumped to the db prompt as well.
I can get the volume to mount read only with zpool import -o readonly=on -f -R /mnt volumename but I am not sure what to do next to get the volume mounted outside of single user mode.

If I install to a new drive and mount the volume in read only mode will I still be able to map ISCSI luns from my vCenter ?

This is a bare metal server, not a VM
No Raid5 is being used, just SATA drives

Thanks in advance

 

[SweetAndLow]

Read the rules.

Hardware specs?

 

[Slydogsz]

The computer was built with spare parts I had laying around
Gigabit motherboard GA-EP45-ud3p
Intel dual core cpu
4gb of ram
4 x 3TB Sata hard drives
1 x 128Gb SSD sata drive
1 x 160GB hard drive

This is just my home nas, I dont have the budget to build with enterprise class hardware, the war department won't allow for that.
I was using it as a ISCSI lun for my ESXi hosts.

 

[SweetAndLow]

Make sure your system meets the minimum specifications then try again. Your problem is most likely from the use of cheap parts or not running maintenance tasks like smart tests or scrubs.

 

[Jailer]

The computer was built with spare parts I had laying around
Gigabit motherboard GA-EP45-ud3p
Intel dual core cpu
4gb of ram
4 x 3TB Sata hard drives
1 x 128Gb SSD sata drive
1 x 160GB hard drive

This is exactly what you don't want to build a FreeNAS box around. That and the fact that you're using half the required minimum RAM it's not surprising that it puked all over itself.

 

[zambanini]

can we throw stones at him? 4gb non ecc ram, l2arc, ...
btw, welcome to the forum : )

 

[Slydogsz]

I knew that when I built it. If it was build for an enterprise it would of gotten the needed ram. It was built for a home lab and the data has minimal value but it would be nice if I could recover it instead of rebuilding . It doesn't see a heavy load, 1 user for a ESXi lab. I built it on the cheep and I recognize this and the risks associated with that. I wasn't really looking for a RCA of why it crashed but was looking for assistance or guidance on how to fix it or if it was fixable.

 

[zambanini]

there is no 'spare part lab' possible with freenas. at least not, when your data is imporant. throw some more ram in it and hope. anyway, no clue why you mentioned solaris.

you need to spend time and knowledge, otherwise freenas is not the right system for you. you did the typical noob mistake.

 

[Jailer]

If it was build for an enterprise it would of gotten the needed ram.

Even if this is just built for a learning experience and to get familiar with FreeNAS, the one thing you can't skimp on is RAM. The problems you are experiencing are exactly why you can't. And you won't get any help from the community when you refuse to at least meet the minimum required hardware specs.

But hey I'm just a nobody who likes hanging out at this place, what do I know. Continue on chasing your tail if you please because that's exactly what's going to happen with your current setup.

 

[Slydogsz]

The reason I mentioned solaris is because that is the top line that I get when the error occurs. It happens on boot up. I have installed freenas on a second drive and when I try to import the pool there I get the same error. Here is a picture of the error. http://imgur.com/qaW3kc2

 

[Robert Trevellyan]

zpool import failure can be due to lack of RAM, so doubling (or better) the RAM might help.

 

[fta]

The reason I mentioned solaris is because that is the top line that I get when the error occurs. It happens on boot up. I have installed freenas on a second drive and when I try to import the pool there I get the same error. Here is a picture of the error. http://imgur.com/qaW3kc2

It looks like metadata corruption. Since the corruption is in the space map, it makes sense that you'd still be able to mount it read only. Something caused some bits to flip, and I wouldn't necessarily put the blame on not enough RAM.

Since you can get it mounted read only, I'm sure there are some recovery options. Hopefully someone who knows that stuff better than me will come along, but as you've seen, everyone is just following the party line of be an a-hole first, be helpful second. Welcome to the forum and try not to let it make you jaded against freenas!

 

[styno]

It looks like metadata corruption.

Agreed, smells like https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193875. What happens when you boot in 9.3? Maybe your system is not 100% compatible yet with the new BSD release.
In any case, you should backup that data from right now (in r/o).

 

[Slydogsz]

It looks like metadata corruption. Since the corruption is in the space map, it makes sense that you'd still be able to mount it read only. Something caused some bits to flip, and I wouldn't necessarily put the blame on not enough RAM.

Since you can get it mounted read only, I'm sure there are some recovery options. Hopefully someone who knows that stuff better than me will come along, but as you've seen, everyone is just following the party line of be an a-hole first, be helpful second. Welcome to the forum and try not to let it make you jaded against freenas!

I understand where they are coming from. If the docs say run on 8gb then you should run on 8gb. I read that but only had 4gb available for this motherboard so I decided to see how 4gb would work. And it worked great for 2 years. I have already found a replacement box that will let me fill it with more ram and will use the ECC ram that I have laying around, so I think I have a decent plan moving forward. I just would like to recover the data from the volume.

So I booted from a new 9.3 install and got the same error when I tried to import. I did manage to get the pool mounted as read only, but I dont see it listed under storage. I ran zpool import -o readonly=on -f -R /mnt volumename and I can see the volume under mnt but when I change to the directory all I see is the jails directory.

How do I get access to the read only volume to back up the data ?

Thanks

 

[depasseg]

I did manage to get the pool mounted as read only, but I dont see it listed under storage.

The GUI doesn't know about it, since it was done at the CLI. It's OK though.

I ran zpool import -o readonly=on -f -R /mnt volumename and I can see the volume under mnt but when I change to the directory all I see is the jails directory.

What is the output of zfs list?

 

[Slydogsz]

Welcome to FreeNAS
[root@freenas] ~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
EllerData 7.22T 443G 140K /mnt/EllerData
EllerData/.system 365M 443G 330M legacy
EllerData/.system/configs-9302fb29893e4327a694006acd742b52 140K 443G 140K legacy
EllerData/.system/cores 33.3M 443G 33.3M legacy
EllerData/.system/rrd-9302fb29893e4327a694006acd742b52 140K 443G 140K legacy
EllerData/.system/samba4 186K 443G 186K legacy
EllerData/.system/syslog-9302fb29893e4327a694006acd742b52 581K 443G 581K legacy
EllerData/Digital 7.22T 4.64T 3.01T -
EllerData/jails 140K 443G 140K /mnt/EllerData/jails
freenas-boot 939M 143G 31K none
freenas-boot/ROOT 931M 143G 31K none
freenas-boot/ROOT/default 931M 143G 931M legacy
freenas-boot/grub 7.77M 143G 7.77M legacy
[root@freenas] ~#

 

[styno]

As you can see by the last char '-' on that line, EllerData/Digital has no mountpoint and thus is not mounted.

Can you also post the output of 'zpool import -nfF EllerData' (instead of the read-only mount) as well?
The '-n' here will not change anything but it will report if the pool is recoverable by a transaction rollback.

Oh and just make sure you have a dd copy of your disks before attempting to do modifications!

 

[Robert Trevellyan]

I just would like to recover the data from the volume.

Do you have SSH working? If so, once you have the necessary dataset(s) mounted, the simplest solution will be SFTP or rsync.

 

[Slydogsz]

As you can see by the last char '-' on that line, EllerData/Digital has no mountpoint and thus is not mounted.

Can you also post the output of 'zpool import -nfF EllerData' (instead of the read-only mount) as well?
The '-n' here will not change anything but it will report if the pool is recoverable by a transaction rollback.

Oh and just make sure you have a dd copy of your disks before attempting to do modifications!

So I ran the 'zpool import -nfF EllerData' command. Looks to be the same output as when I run the command with just a -f argument. Lots of text scrolls by to fast to read and then it does a restart.

I was talking with some storage experts today at work and it was suggested that I might have an issue with my SSD cache. The suggestion was made to pull the SSD cache drive out of the volume and reboot the nas. How much risk is there in doing that ?

Do you have SSH working? If so, once you have the necessary dataset(s) mounted, the simplest solution will be SFTP or rsync.

Yes I have ssh working. I am not real clear on how to get the dataset mounted to even start backing it up from the read only mount.

 

[Robert Trevellyan]

The suggestion was made to pull the SSD cache drive out of the volume and reboot the nas. How much risk is there in doing that ?

I doubt it will help, but it probably won't make it any worse. You can always put it back.

Another member with pool mounting problems and insufficient RAM was able to recover by moving all the drives to a different box with more RAM.

I am not real clear on how to get the dataset mounted to even start backing it up from the read only mount.

https://www.freebsd.org/cgi/man.cgi?zfs(8)
Hint: search the page for 'mount'.