Server Locks up with "Fatal Trap 12"

[exitcomestothis]

I've been running FreeNAS 8.2.0-p1 64bit since it was released, and have never had a single problem with it. Always been rock solid and stable, with the exception of the occasional thumb drive failure.

Over the last 2 weeks however, my luck's changed on that front. It seems that every 18-36hrs or so, my server will completely lock up and display "fatal trap 12: page fault while in kernel mode" I've attached a couple screenshots below.

When the crash happens, it's always during a time when the system is idle, and I've noticed a pattern that it likes to crash around 7:00-7:30am. Looking at the memory graphs, there is still memory free, but the usage pattern looks a bit odd.

Steps I've taken to address the problem:
Replace the usb drive and re image.
Re-image usb drive plus move stick to a different usb port
Re-image usb drive and manually resetup shares, accounts, etc.
Ran memtestx86 for roughly 56hrs, and 0 errors were found.
Both drives have passed a smart test (short version).

I setup a syslog server in hopes to capture an error of some sort that's hapening before it crashes, but don't see anything suspicious. I've attached this log as 2.9.15.txt.


Specs on my system:Mobo: Intel D975XBX2KR Mobo
CPU: Core 2 Duo 6300
RAM: 6gb ram (2 2gb sticks and 2 1gb sticks)
HDD: 2x 3TB WD Red (mirrored with ZFS v15)
NIC: Intel EXPI9301CTBLK (pci express)

System has 4 SMB shares, and 1 NFS share. No tunables or advanced config settings have been set.

Does anyone have any suggestions on things I should be trying to remedy this?

 

[Apollo]

What is the status of your pool?

 

[exitcomestothis]

Pool status is healthy. Server locked up again with this error just a little bit ago at 7:35am pst.

pool: Vol1
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Vol1 ONLINE 0 0 0
mirror ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors

 

[Apollo]

You have never run a scrub on your volume. When did you create the pool?. I don't know if it is wise to do it now.
But it could be possible there is something fishy about your drives.

 

[exitcomestothis]

I ran a scrub about a month ago, and everything came back clean at that point. Not sure why it's not showing up on the status output. Would my reimaging the usb stick have anything to do with it?

The pool was originally created back in July of 2012 (+/- 1 month or so).

 

[gpsguy]

Usually these errors h are nindicative of a hardware issue. You are a little short on RAM. Run 3 complete passes of Memtest86+ on your server and see if it finds any problems.


Sent from my phone

 

[exitcomestothis]

I let memtest run over the weekend (last weekend I believe), and it didn't find any errors. I can't remember how many passes it logged, but it was definitely more than 3.

No errors were found.

 

[gpsguy]

I ran into this problem in June 2012. It started happening out of the blue. Prior to that I didn't have any problems. After a lot of troubleshooting I gave up and bought new hardware. I moved the disks and flash drive to the new hardware and haven't had a problem since then.

Sorry, I can't be more helpful.


Sent from my phone

 

[Apollo]

Do you have autotune enabled? If not then enable it.Do you have any type of jails installed?
Any cron jobs you have enabled?

 

[cyberjock]

Lookie there. You are running 8.2.0 and so was this guy... https://forums.freenas.org/index.php?threads/fatal-trap-12-page-fault-while-in-kernel-mode.10268/

Same error and same old version. ;)

 

[exitcomestothis]

Nope, no jails, no tunables/sysctls, and no cronjobs. Everything is default, with the exception of the SMB shares and user accounts.

I went ahead and enabled autotune, and see that it's added a few tunables and sysctls. Will try this and see how it goes.

It's also worth noting that the bios is at it's most current version (the same version that was on it when I first started using FreeNAS), and there have been no bios updates released since.

 

[exitcomestothis]

After enabling autotune, I noticed now that the system will lock up after about 8 or so hrs. I've disabled autotune and removed all sysctls and tunables, and am looking at ordering new hardware.

 

[Apollo]

After enabling autotune, I noticed now that the system will lock up after about 8 or so hrs. I've disabled autotune and removed all sysctls and tunables, and am looking at ordering new hardware.

In the time being why not upgrading to 9.2 instead? Cyberjoke has included a link to a thread that pointed an issue under 8.2.

 

[exitcomestothis]

I was considering that, but decided against as the docs for 9.2 seemed pretty adamant that 8gb was the minimum required. Though 8.2 docs recommended 8gb, 6gb was the minimum for my setup.

Would it be wise to rub 9.2 with only 6gb?

 

[gpsguy]

Users (myself included) have run into that error with other versions of FreeNAS. The OP's hardware isn't recommended and he's short on RAM.

CPU: Core 2 Duo 6300
RAM: 6gb ram (2 2gb sticks and 2 1gb sticks)

If he's willing to buy new hardware, I'd recommend going that route.

In the time being why not upgrading to 9.2 instead? Cyberjoke has included a link to a thread that pointed an issue under 8.2.