Cannot Import : no such pool available

[David3D]

Hey guys,

So I've got a bit of a problem.

I have an old FreeNAS box which I was using for testing/fun purposes and junk files which I believed had no important data on it. Of course that is what I thought until I couldn't find what I wanted one day and figured it must be on the old box.

Version I'm using: FreeNAS-9.1.0-RELEASE-x64 (dff7d13)

First I saw that Alert System was telling me that the volumes I wanted to access was "UNKNOWN". I tried rebooting and after a few reboots, instead of defaulting to unknown, it just wouldn't boot with the affected drives attached and instead telling me the following:

'Mounting local file systems:.
cannot import '8582060359732297958' : no such pool available'

Then continues on to say
ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Error 5, Retries exhausted
...etc...etc

I don't know exactly what is important in order to best give everyone the best information. I've tried sourcing logs from /var/log/dmesg.yesterday but it does not seem to contain what I saw on the console screen. I did however, video it if that helps, haha :)

I've got a feeling the data is gone, since one drive is probably dead. They were not striped. But I believe the configuration was a mirror on a hardware RAID. The server wasn't exactly at spec either.

I know all this now of course, other important data is on a much better server that is running flawlessly with regular 60 minute backups..etc, which is why this server was retired, because I relised I was freakin' stupid. Obviously I missed the data mirrored across two 500GB drives and here we are.

I completed a search and I might be blind, but I could not find my specific error. Perhaps my problem manifests itself in different ways, or what I've posted is of no help at all...But I can't seem to see any road to recovery at this point.

I have both drives free from the server and can plug them in where ever I wish. I've seen posts regarding ZFS recovery scripts (experimental ones) and posts on using Linux to recover bad ZFS pools.

I basically just need a direction to follow. Anything I should or shouldn't do with the drives? Any live CD recommendations for recovery of ZFS pools/volumes?

I'm probably just being an ignorant n00b thinking I'll be getting any of it back. Any more information you need, it's yours! If I knew what info you needed for this issue it would be included, but I'm totally green!

Any help at all is appreciated.

Cheers guys!
David

 

[cyberjock]

So provide the following with as many "good" drives as you can get the system to boot up with:

1. debug (can be gotten from the WebGUI)
2. dmesg output (pastebin is good for the rest)
3. zpool status
4. zpool import
5. Your actual hardware.

But, being that you chose to go with hardware RAID, you *did* probably lose your data. But we'll see...

Don't go trying to do anything with ZFS either. Just be patient and we'll see what your status is.

 

[David3D]

So upon further analysis, it looks as if bad ECC RAM may have grenaded something. The board is now segregating 5 out of the 6GB of installed memory. Looks like something funky may have happened to the board otherwise...FreeNAS is now only reporting 1004MB of memory. This is what I get for old hardware I guess. At least I called it happening based on information here. Too bad I missed data!

...This should be fun.

Processor: Intel(R) Xeon(TM) CPU 3.00GHz
Memory Was 6GB of DDR ECC memory
Intel motherboard (Most probably Intel S5000PAL)
2x 2TB WD RED drives
2x 500GB SEGATE drives (possibly dead or bad zpool at least)
1x 250GB Toshiba disk for dumping crap
1x 500GB Toshiba disk to extend crap dumping
Redundant 700W Power supplies
It's essentially an old Intel SR2500 2U unit that was laying around. As mentioned above...Well below spec which is why it was taken out of production.

freenas-debug:
http://pastebin.com/0c9KYzf5

zpool import - Does not seem to do anything. Is there
zpool status:

Code:

NAME                                            STATE     READ WRITE CKS
UM                                                                          
        MEDIA_DRIVES                                    ONLINE       0     0
0                                                                          
          mirror-0                                      ONLINE       0     0
0                                                                          
            gptid/e4b87da1-1667-11e3-b247-000423b80262  ONLINE       0     0
0                                                                          
            gptid/e55ddabc-1667-11e3-b247-000423b80262  ONLINE       0     0
0                                                                          
                                                                            
errors: No known data errors                                                
                                                                            
  pool: SWAP                                                                
state: ONLINE                                                              
  scan: scrub repaired 28K in 0h16m with 0 errors on Sun Jul 27 00:16:17 2014
config:                                                                     
                                                                            
        NAME                                          STATE     READ WRITE CKSUM
        SWAP                                          ONLINE       0     0     0
          gptid/e0f79237-15e5-11e3-8d87-000423b80262  ONLINE       0     0     0
          gptid/e20427cd-15e5-11e3-8d87-000423b80262  ONLINE       0     0     0
                                                                            
errors: No known data errors

I can't seem to find the file location of the dmesg output. Unless I'm supposed to copy it directly from the Shell, which seems impossible, since there is no scroll functionality. I have root access via FTP for any configuration files that may be required. I've done a search through the documentation regarding the location of the dmesg output as well as a search here on the forums and came up empty.

EDIT: I havn't tried it yet, but if FreeNAS will accept drives inserted after boot, I guess I could try that. I don't remember setting up a hotswap functionality however, which may be a problem.

Cheers,
David

 

[danb35]

Well, if you go to the shell and do

# dmesg > /mnt/SWAP/dmesg.txt

that will save the dmesg output into a text file (in this case, /mnt/SWAP/dmesg.txt, but it can go anywhere) that you can then download, upload, paste, or whatever. I'll note that even 6 GB of RAM doesn't quite meet the system requirements, but I wouldn't expect that would be your problem.

 

[David3D]

Well, if you go to the shell and do

# dmesg > /mnt/SWAP/dmesg.txt

that will save the dmesg output into a text file (in this case, /mnt/SWAP/dmesg.txt, but it can go anywhere) that you can then download, upload, paste, or whatever. I'll note that even 6 GB of RAM doesn't quite meet the system requirements, but I wouldn't expect that would be your problem.

Hi dan,

Thanks for the info. I'm not familiar with all the commands either. :( The server started out on 2GB of RAM then 4 and then 6. I then learned that was FAR too little, which is why it was almost immediately retired and a new solution was implemented. It became my playground and I broke it.

dmesg information pasted here: http://pastebin.com/Ye7wi8zU

Cheers!

 

[david kennedy]

Well, if you go to the shell and do

# dmesg > /mnt/SWAP/dmesg.txt

that will save the dmesg output into a text file (in this case, /mnt/SWAP/dmesg.txt, but it can go anywhere) that you can then download, upload, paste, or whatever. I'll note that even 6 GB of RAM doesn't quite meet the system requirements, but I wouldn't expect that would be your problem.

There are a number of examples of "bad things" happening when ZFS has too little memory to work with.

If memory serves, back in the old Solaris days you could get pools which once they crossed a "magic" threshold of committed data they would no longer import until you added ram. Said another way, pool was fine when it was empty but as it started filling with data would reach a point where it would not mount due to insufficient memory.

Not sure how another OS using the ZFS code base would handle this situation, this could be the source of some of these "corruption" issues.

 

[David3D]

There are a number of examples of "bad things" happening when ZFS has too little memory to work
Not sure how another OS using the ZFS code base would handle this situation, this could be the source of some of these "corruption" issues.

It could well be! I've checked other pools for data we may or may not need and it seems these drives were the only ones that contained anything important. Any methods for recovery are possible. I know it's not as simple as plug it in and browse around like you might with a Windows machine. I'm not sure that is the case with a Linux based OS that supports the FreeNAS filing system.

Cheers.

 

[cyberjock]

Yeah, your 1GB of memory most certainly trashed your box. I'd wager that if it wasn't the 1GB of RAM that was the problem the fact that your system suddenly had some kind of hardware failure that lead to only 1GB of RAM being used may be the problem. Perhaps a bunch of trash data was written to the pool when this failure took place?

Of course, if you did use a hardware RAID controller with the mirror that could be the problem too.

In any case its not too likely you'll get your data back. If you have backups you should definitely get them out. ;)

This is kind of why we push so hard for the right hardware. If things go wrong there's little or nothing you can do, and the cause is kind of guess work.

 

[David3D]

Yeah, your 1GB of memory most certainly trashed your box. I'd wager that if it wasn't the 1GB of RAM that was the problem the fact that your system suddenly had some kind of hardware failure that lead to only 1GB of RAM being used may be the problem. Perhaps a bunch of trash data was written to the pool when this failure took place?

Of course, if you did use a hardware RAID controller with the mirror that could be the problem too.

In any case its not too likely you'll get your data back. If you have backups you should definitely get them out. ;)

This is kind of why we push so hard for the right hardware. If things go wrong there's little or nothing you can do, and the cause is kind of guess work.

I essentially binned the box for this very reason.

Just to make sure...If I check to make sure the drives aren't toast to begin with (I havn't plugged them in incase there is something I don't know about) and check that they are actually capable of running, connecting to a controller...etc...How do you suggest I go about accessing a pool outside of a FreeNAS environment to poke it with a stick? Or is this technically impossible? None of the pools were encrypted or anything like that.

EDIT: I have a backup, but it's a bit older than I'd like! Since the box was sidelined, none of the information was being actively backed up unfortunately.

 

[cyberjock]

If they drives aren't toast I'd try to do a pool auto-import from a FreeNAS machine. It likely won't mount on linux because linux doesn't support all the feature flags that FreeNAS/FreeBSD uses.

Other than that you are going to be in for a world of hurt. As my noobie presentation says there aren't any ZFS recovery tools. If your data is important and you just gotta have it back I offer recovery services, but it isn't cheap and there's of course no guarantee your data can be recovered.

Alternatively you can try reading through the internet and trying things. But to be honest I haven't seen a pool be recovered from problems related to low RAM. So I tend to think that you need to just write off the data. :(

 

[David3D]

If they drives aren't toast I'd try to do a pool auto-import from a FreeNAS machine. It likely won't mount on linux because linux doesn't support all the feature flags that FreeNAS/FreeBSD uses.

Other than that you are going to be in for a world of hurt. As my noobie presentation says there aren't any ZFS recovery tools. If your data is important and you just gotta have it back I offer recovery services, but it isn't cheap and there's of course no guarantee your data can be recovered.

Alternatively you can try reading through the internet and trying things. But to be honest I haven't seen a pool be recovered from problems related to low RAM. So I tend to think that you need to just write off the data. :(

As expected. I've told everybody that there is no hope.

Since the FreeNAS box refuses to boot with the drives in (going in a loop with trying to call the drives and get a responce) is there a way I can simply plug them in while it is running and try pool auto-import...Or is there some mechanism that is activated on boot to detect the drives then? Otherwise, I will try a fresh FreeNAS installation on a separate USB drive and try auto import on there.

Cheers.

 

[cyberjock]

If the system is rebooting when booting up it's likely trying to import the pool and then crashing. If that's the case then auto-importing isn't going to gain you anything as it's nothing more but an import of the pool on a FreeNAS configuration that doesn't recognize the pool yet. In your case it sounds like auto-importing is going to give you the same thing, you'll import the pool and the box will immediately crash and reboot.

 

[David3D]

If the system is rebooting when booting up it's likely trying to import the pool and then crashing. If that's the case then auto-importing isn't going to gain you anything as it's nothing more but an import of the pool on a FreeNAS configuration that doesn't recognize the pool yet. In your case it sounds like auto-importing is going to give you the same thing, you'll import the pool and the box will immediately crash and reboot.

It's not rebooting as such. Just saying it's exhausted so many retries and then retries again anyway. Then tells me the pool is unavailable and refuses to move past that stage and allow me to access other pools on the box.

Not sure if there was a way to force it. But I guess if I booted the box, plugged the drives in to their respective slots and tried the import command, it might be the "force it" option that I am looking for.

I shall report back with results!