Diagnosing multiple causes of Fatal Trap 12

[xzachtli]

I request assistance gathering more details from a persistent crash I am seeing on my setup in hopes of finding a solution or submitting a helpful bug report.

I have the following setup

• FreeNAS-9.2.1.5-RELEASE-x64
• Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz
• SuperMicro X9SCM-F
• 24 GB ECC RAM
• Intel Pro/1000 NIC

My only ZFS pool was created in ZFSGuru (to support 4k sector drives - ashift: 12) and remains at v28. I ran in Solaris Express 11.11 with napp-it for 2 years and recently imported the pool to FreeNAS

• raidz2 - 6 x 3TB Hitachi
• raidz2 - 6 x 3TB Hitachi
• raidz2 - 6 x 2TB Seagate
• cache - 240GB OCZ SSD

I have to mount the pool in readonly mode in CIFS to not have constant crashes (crash detail below). I assumed it was just a Samba issue and was waiting out 9.2.2 for a potential fix. Note: I didn't have to actually do anything (move/delete files) over SMB, just have it mounted as rw to crash.
Recently, I added the sabnzbd and sickbeard plugins. Both work, but ended up causing what looks like the same crashes I had received when running the CIFS shares in rw mode. After debugging, I can run the sabnzbd plugin just fine - downloading anything is no issue. It appears to be the sickbeard plugin that generates the crash, specifically the process that scans for completed downloads, copies to the media directory, and renames them. This does not operate over CIFS/SMB, but instead uses the mounted pool through the jail storage for direct file access.
Crash details: (As many as I can get - help requested for more)
May 4 15:52:28 freenas kernel: Fatal trap 12: page fault while in kernel mode
May 4 15:52:28 freenas kernel: cpuid = 0; apic id = 00
May 4 15:52:28 freenas kernel: fault virtual address= 0x48
May 4 15:52:28 freenas kernel: fault code= supervisor read data, page not present
May 4 15:52:28 freenas kernel: instruction pointer= 0x20:0xffffffff81b133a0
May 4 15:52:28 freenas kernel: stack pointer = 0x28:0xffffff869ea793a0
May 4 15:52:28 freenas kernel: frame pointer = 0x28:0xffffff869ea79430
May 4 15:52:28 freenas kernel: code segment= base 0x0, limit 0xfffff, type 0x1b
May 4 15:52:28 freenas kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
This is what I get repeating over and over for a few minutes in the console until the machine restarts:

I have since disabled the sickbeard jail and remain in readonly CIFS mode
Things I have tried:

• Updated from FreeNAS 9.2.1.4 (where I started)
• Change SMB min and max to 2.0
• Enable debug kernel and check /data/crash - no files

Other maybe useful details:

• Services: CIFS, iSCSI, NFS, SMART, SSH
• Memory utilization does not peak prior to crash
• Running FreeNAS on SanDisk Extreme 16GB USB3 drive (internal USB 2.0 port on MB)

Thanks for any additional steps you can recommend to get more data.

 

[cyberjock]

Few things that may or may not be your problem.

Your 240GB L2ARC is 10x the quantity of your system RAM. That's not going to work properly and isn't even recommended. You should never exceed 5x your ARC, which is likely peaking at about 20GB max. Additionally, I've been around and around with various users in the forum over this, but the manual isn't kidding when it says "install the maximum allowed RAM before considering an L2ARC". The reason.. because internally to ZFS you need lots of RAM before you can start using an L2ARC without potential problems. Additionally, I constantly tell people not to consider an L2ARC until you have 64GB of RAM. Yes, I know you're about to tell me "but but but my motherboard can't go that high". Well, I have the same board.. and I'm not using an L2ARC either.

Importing pools from other OSes has been a problem. Not sure why, but it is. For that reason we tell people not to import pools from other OSes. If you need to use FreeNAS create a new pool and migrate your data over. The issue with it crashing seems to manifest itself at 2 times for various people.. on bootup/mounting of the zpool or when accessing the zpool. Don't have much more info because everyone that just creates a new pool and migrates their data over sees the problem go away. ;)

 

[xzachtli]

cyberjock-

Thanks for the quick response! I have no issue removing the L2ARC. I had installed it years ago when I was new to ZFS under the assumption that it would fill with the most often used data and take load OFF the RAM since I only had 8GB at the time and R/W performance wasn't of concern to me. I will max the board out with RAM as finances allow in the near future.

I have tried to reset all of the zfs options for my pool/datasets including sharesmb and sharenfs to defaults to hopefully prevent any issue from the import. This did not change the outcome. My stopping point is I have no place to put the data to be able to destroy/recreate the pool. I was also hesitant to use a pool version other than v28 should FreeNAS not work out for me (although I already love it). I don't suppose the reason recreating the pool directly under FreeNAS works better because the pool is then at v5000? Does FreeNAS expect feature flags to be enabled on the ZFS pool nowadays?

 

[cyberjock]

No. I'm still on v28 and I recommend people stay on v28 whenever possible.

This isn't a feature flags problem. This is more like FreeNAS has expectations that aren't being met.

FreeNAS expects you to have created the pool from the WebGUI. Because you are *supposed* to do it that way it creates the pool with certain characteristics on the hard drive itself. You obviously won't meet those characteristics with a pool that wasn't built by FreeNAS. Unfortunately I don't know how to solve the problem so the only way I know of to deal with it is to create the pool on FreeNAS to begin with. :(

 

[xzachtli]

I have removed the L2ARC from the pool and want to fire up my plugins again to test but would love to be able to capture an actual crash dump, as I am expecting it to crash again. Is there anything I can do to ensure I can get one aside from enabling the debug kernel like I have already attempted?

 

[cyberjock]

There is, but I don't remember how off the top of my head. There's a way to upload the crash to iXsystems from a WebGUI settings and then ask a developer to comment on it. I don't know exactly what they need to find your crash from the tons of other crashes they get daily. You could pop in IRC and ask though.