Fatal Trap 12 - In use by another user

[rvo500]

System: 32 bit motherboard, carrying a 3.2 GHz Intel processor, 4 GB RAM.
Version: FreeNAS 9.2.1.9 x86
Drives: (2) qty of 4TB SATA
Setup: SATA1_VOL1 using both the drives, as a mirror, with a 3.47 TB ZPool set up, so SATA1_VOL1/ZVOL1
The z volume is then presented up via iSCSI.
General description: Very light use SAN presenting storage to VMware via iSCSI.

Disclaimer: Before I get clobbered for 32 bit with 4GB of RAM, I've had great success with this given the very light duty I generate. I have no reason to believe it is the source of my issue - which I will explain.

Problem: Using VMware Storage vMotion, my production virtual server, carrying live data (ie critical to me) was migrated from another FreeNAS 9.2.1.9 machine presenting iSCSI, with identical hardware except for (2) 200 GB SATA drives. This process was successful overnight. The typical caveat is no backups to speak of - but it worked great. The next day, I went to migrate VM #2. The migration started nicely, and data was flowing well. Given I was more interactive during the day, so I got to looking over other things and getting my ducks in a row, doing cleanup, etc. During the migration (which was stupid), I believe I triggered a crash - by the change of the default FreeNAS host name to my choice of host names. I've tried to trace back my steps, and this truly seems like the crash candidate, because I did do this, and it was about the same time.

Current symptoms: When I try to boot, I get a Fatal Trap 12 error, and my zpool will not mount, and leaves me with a DB> prompt. In other words, my priceless data is held hostage. It does indeed say that the pool is in use by another user.

Troubleshooting attempts: I've Bing'd the daylights out of this. No luck. But I have used tactics shown for other errors. I've tried many things, and narrowed it to this (somewhat promising) result. If I boot, choose option 5, and get my OK prompt - then:

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot -s
zpool import -o readonly=on -f <poolname> (best of my recollection at the time - basically mount the pool read only)

Then my pool mounts, and a zpool status shows everything clean. No corruption. It's clean.

Given I can mount the pool read-only in single user mode, and the pool shows clean - how can I get my data back? As always, this data is critical and irreplaceable. I can't do a full regular boot - all I get is the fatal trap 12 crash. Read-only is my only functional mounting option, and I don't know what I can do with it once mounted. It does show clean. I just don't know how to use it, or what I can do to make the system return to function in regular full boot mode.

 

[jgreco]

This is typical of the problems people experience when they've not given FreeNAS the required RAM. I raised the minimum required RAM from 6GB to 8GB years ago because of this. You are welcome to your opinion as to whether or not 4GB is enough, but we know it not to be, especially for uses such as iSCSI. Screwing your pool and totally losing it is not unexpected. You can consider yourself lucky that you can mount it read-only, manually configure networking, and then netcat the data off... but it is quite possible you'll find a panic in the middle of that process.

 

[gpsguy]

I read this earlier and cringed at the thought of trying to run this setup on 4GB RAM.

If you read any of jgreco's stickies regarding iSCSI on FreeNAS you'd know that the minimum recommendation for RAM would be 64GB.

He's got a box with about 7TB of usable space and recently upgraded it from 128GB to 256GB RAM.


Sent from my phone

 

[rvo500]

I do have my opinion on the 4 GB topic, given the years of success I've had with it, and given the nature of what I believe triggered my issue. I believe I self-inflicted my issue with the host name change during a migration (dumb move). I do, however, given all the feedback I've read on the topic, intend to *ONLY* move forward with 8GB+ on 64 bit. In other words, once I get my data back, I'm done with 32 bit and 4GB machines as fast as possible. It's just not worth it any more. At this point it is a risk I'm not willing to continue. There is too much research-backed data dissuading folks from operating this way, even if it seems OK. So, understood on the 4GB topic. That being said...

I would think there would be traps regarding the host name change, to prevent this - but I'm fairly certain the host name change during an active data move was my crash source in this case.

I have good hopes that I can, given my clean read-only status, recover my data. (Fingers crossed of course). I'm not sure how to configure FreeNAS networking from single user mode, or to use netcat as you described. Google and RTFM isn't helping much here. Do you have a suggested method, set of commands, etc? Are there alternate, more easy methods? Can I use an OpenIndiana live boot and set of commands - to where I can RO the zpool and attach a USB drive and copy the data over? I'd like to keep it as straight forward as I can. Given the countless hours of frustration and hopelessness I've put into this - I'd just as soon have a friendly method that doesn't take much additional learning or risk.

 

[jgreco]

I read this earlier and cringed at the thought of trying to run this setup on 4GB RAM.

If you read any of jgreco's stickies regarding iSCSI on FreeNAS you'd know that the minimum recommendation for RAM would be 64GB.

He's got a box with about 7TB of usable space and recently upgraded it from 128GB to 256GB RAM.

That's enthusiastically doubled on all fronts. My minimum recommendation would be 32GB, and that's mainly because there's a large number of platforms out there that max at that. And I just upped storage3 from 64GB to 128GB. But you're at least wrong in a good direction. ;-)

As noted elsewhere, the ARC impact of block storage argues for a lot of RAM. 8GB is the bare minimum requirement FreeNAS has for NORMAL LIGHT FILESERVER USAGE, and block storage is anything but normal light fileserver usage.

 

[jgreco]

I do have my opinion on the 4 GB topic, given the years of success I've had with it, and given the nature of what I believe triggered my issue. I believe I self-inflicted my issue with the host name change during a migration (dumb move).

Yes, probably a dumb move since it probably restarted networking, but then what happened next has happened to lots of other people. People who didn't taunt their system in that manner. You were playing with fire, whether or not you believe it.

Now, I've been here doing this forum support thing for about five years now, and I was the person that correlated a lack of RAM as being a pool killer back in the ~2013 era. I used to feel sorry for the victims, because, honestly, no one had warned them. I can't say I feel so sorry for a lot of the people who've shown up since then. Many of them felt that their circumstances were special, or their use model wasn't demanding, or that the fact that their machine was 32 bit and couldn't do more than 4GB meant that they should receive special dispensation from the minimum requirements.

I finally got sick and tired of the crap. It's 2016. There's no reason for this. So, earlier this year, I started actually collecting a list of failures. Because I am so very fscking tired of explaining to people like you that you are not the exception and this happens all the time. So far my list isn't that big, because we're actually pretty good at guiding people the right way, and I didn't care to actually dig back through five years worth of threads. But it was easy to find a few recent examples.

https://forums.freenas.org/index.ph...-guide-to-things-not-to-do.40924/#post-259708

So hear my words when I say I've seen it more than a hundred times before.

As for recovering the data, you can choose to manually configure networking (which requires some idea as to what your networking device is, the local network configuration, etc ). Basically if you had an older Intel card and a network of 192.168.2.0/24 where the NAS was at .10, it might be something like "ifconfig em0 inet 192.168.2.10 netmask 0xffffff00". Then you could probably create a new device on a better system and use netcat to copy the data over; I don't have an incantation handy because this is kind of dodgy. The other thing you could do is to actually move those disks over to a new NAS that had another datastore, import the pool by hand, and then use a local dd copy against the zvols to copy the data off them.

 

[DrKK]

Here's some Fatal Trap:

https://www.youtube.com/watch?v=5-2nByd2cr4

 

[jgreco]

iSCSI on a 4GB FreeNAS. I like it. We should archive this in that thread we started a couple week ago!

We did. Feel free to follow up to that thread with any other examples as appropriate. For five years I've been keeping mental tabs on trends and issues, which has formed the basis for a lot of the BCP advice we hand out, but my failure to maintain an actual list has made it all too easy for users to be dismissive because I couldn't point them at the stack of fails that I know I've seen. That seems particularly relevant to the memory size and ECC issues but others as well.

 

[anodos]

Here's some Fatal Trap:

https://www.youtube.com/watch?v=5-2nByd2cr4

A++++ Great seller! Would recommend!

 

[m0nkey_]

Here's some Fatal Trap:

https://www.youtube.com/watch?v=5-2nByd2cr4

When I think fatal trap, I think Sharon Stone.

 

[jgreco]

When I think fatal trap, I think Sharon Stone.

Fatal something, anyways.

 

[Ericloewe]

Here's some Fatal Trap:

https://www.youtube.com/watch?v=5-2nByd2cr4

+1 for originality.