9.2 Upgrade from 9.1.0 Release Various Fatal trap 9's

[Shdw]

Let me start with saying hello and thank you for a wonderful open source project. I am new to Linux and FreeNAS, so I'll do my best to give useful information. Please help!

System/Background info:

I'm running a gen 4 core i5 with 8gb ram and a 2tb hard drive connected to sata0. FreeNAS 9.1.0 release was originally installed on an 8 gig thumb drive hooked into the bios flashback usb slot, a USB3.0 connection. This is the motherboard. The first instal attempt was made on one of the other USB 3.0 connections, but failed to work for reasons I cannot remember. For some reason it worked on the flashback usb3.0 and then after install I moved it to another USB3.0 spot with no hangups and proceeded to setup through the GUI.

Setup included 5 users, 6 groups, a ZFS dataset for each of the 5 users, activation of apple shares, windows shares, SMART enabled, and ftp (w/o tls) setup successfully. All on an unencrypted ZFS pool on the 2tb drive attached ada0. After hangups with enabling tls I came across Feature #1910 in my google searches. Seeing that there were updates to TLS options in 9.2.0 release I promptly went about updating. <Enter Fatal trap 9 errors>

Fatal 9 Error 1

This error being 3 days ago I don't even remember the thread pid or any other useful information. All I can tell you is I updated via the GUI to 9.2.0 and got the error on reboot. Realizing after a few hours that USB3.0 was no longer supported. Some more searching brought me to this article whereby I rebooted to a USB2.0 slot and was able to get into the GUI and implement tunable xhci_load in accordance with the article. Needless to say that didn't work. Is there a way to enable USB 3.0 again?

Fatal 9 Error 2

Again, this one was two days back and I don't remember the details of the error. It occurred after doing a fresh install back on a USB 2.0 connection. That went without a hitch and brought me to Auto import volume. No bueno, Fatal trap 9 error that I don't recall. Unable to recover this forced a reinstall and no more attempts to Auto import; instead I went straight to wiping the drive and starting over.

With no real data saved on the drive yet this seemed the simplest solution.

Fatal 9 Error 3 and 4

Finally we arrive at today and another few hours of fatal 9's, two more fresh installs and no luck. Today's fatal 9's were thread pid 259 following a lock up after trying a wipe to all zero's on ada0. This last one, a thread pid 257 tid 100085, follows a successful quick wipe of the drive and then a failed attempt to mount a stripe zfs volume.

At this point I cannot boot, and in no case with any of these trap 9's was I able to successfully reboot. This trap 9 will continue until I fresh install and repeat the headache. Is there anything I can do? Anymore info I can give you? Currently I'm at 'db>' command line following the trap 9 and am able to type help and various other commands, not that I've a clue what any of it means. :)

Thank you again for the program and any help would be greatly appreciated.

Regards,

Brian

 

[warri]

Is there any reason you need to enable USB 3? Otherwise just leave it running in USB 2.0 mode.

Random kernel faults may indicate a hardware problem. I suggest you to run memtest to verify that your RAM is ok. My second suggestion would be to try a new USB stick.

 

[cyberjock]

I'm not 100% sure if the cause of your problems, but here's some stuff that makes me cringe...

1. You really should be using ECC RAM with FreeNAS. Might not be your problem(a RAM test could help rule this out).
2. Your motherboard is a desktop board. That being said you've got extra hardware on it that is definitely not compatible with FreeNAS(audio , ASUS EPU, and the other Asus exclusive stuff definitely comes to mind). Asus(and the other high end desktop motherboard manufacturers) include extra bells and whistles to make potential buyers pick their hardware. But guess what? That stuff is exactly what you may want for a desktop, but is exactly what you don't want on a server. It's stuff that could go wrong, and if it has a problem and tries to report an error to the hardware and there's no driver there's no telling what the consequence could be. This is one of many reasons why we recommend server-grade boards for FreeNAS.
3. USB3 is a mess for FreeBSD. It's disabled in FreeNAS 9.2.0 by default. This is recommended and preferred. Considering you are already having problems I wouldn't use the USB3 ports and I wouldn't try to enable USB3. It doesn't appreciable affect your server performance unless you are using a bunch of USB hard drives in your pool(and if you did you'd have MUCH bigger problems).

You could try checking for obvious stuff, BIOS update, reset BIOS settings to default, try disabling hardware or changing settings to see if anything helps.

 

[Shdw]

Is there any reason you need to enable USB 3? Otherwise just leave it running in USB 2.0 mode.

Random kernel faults may indicate a hardware problem. I suggest you to run memtest to verify that your RAM is ok. My second suggestion would be to try a new USB stick.

For now, USB2.0 will suffice. However, down the road when actual drives are attached the data transfer of USB 3.0 would be nice to have.

RAM swapped to my other build and ran without a hitch. I don't have another stick handy, I'll see if I can dig one up and give that a shot. Is there a simpler way to test this stick, perhaps on one of my windows or mac machines?

FWIW It installs, boots, and allows manipulation of various GUI options without a problem. Since sticking with USB2.0 the fatal 9's only present as soon as I try to mount the volume. I've been trying to find a way to reformat the volume via command line, but my search has been futile.

 

[warri]

I'd still run a memory test to make sure the RAM is OK. I'm afraid that I don't know of any easy way to validate your USB stick. Also as cyberjock pointed out, make sure you are running on the latest firmwares / BIOS.

Can you mount the volume on another system? We had cases of irreparably damaged pools which caused system panics on mount.

Usually FreeBSD gets proper support for new hardware, but it might take a while. Maybe there will be better USB 3 support with FreeBSD 10 or 11.

 

[Shdw]

Can you mount the volume on another system? We had cases of irreparably damaged pools which caused system panics on mount.

I'll go check bios firmware and such. The volume mounted fine on the 9.1.0 release. The issues all came after updating. Perhaps trying to go back to 9.1.0 would be worth a shot?

Is there a simple way to run a mem test via the bios or from command line. I'm only familiar with mac/windows. And stupid me for not having ECC ram, bought it new too. :(

This is the ram: http://www.newegg.com/Product/Product.aspx?Item=N82E16820148544 Pretty sure it isn't ecc though.

 

[warri]

You can put a bootable memtest86+ image on any CD or USB stick. Then reboot the system and let it run for at least 3 passes (takes several hours). You can get the ISO or images here: http://www.memtest.org/

Of course you could also go back to 9.1.0 and test the import. I'm just stressing about the memory test because you could damage your pool when trying to operate/mount it with faulty RAM.

 

[cyberjock]

There's a chance you've damaged the pool from all of the crashes too...

 

[Shdw]

You can put a bootable memtest86+ image on any CD or USB stick.

Dumb simple google question, sorry. Already done, test running.

 

[Shdw]

There's a chance you've damaged the pool from all of the crashes too...

That's what I was afraid of. Is there a way to fix that, I thought reformatting the disk from command line might solve that.

 

[warri]

Yes, you can wipe the drives and recreate the pool if it is damaged. If you are unfamiliar with the CLI, there are tools like DBAN (again, bootable image) which are easy to use.

 

[Shdw]

Yes, you can wipe the drives and recreate the pool if it is damaged. If you are unfamiliar with the CLI, there are tools like DBAN (again, bootable image) which are easy to use.

Thank you. There is no way to do it from command line? Memtest at 60 percent, so far so good, will get this image up and ready next. BIOS updated. Thanks again for all the help gentlemen/ladies. :)

 

[Shdw]

Memtest was fine. Reformatted thumb drive to do bios flash and reformatted again (FAT). Now doing DBAN to wipe the drive, looks like a long process. Will post up when it's all done and let you know how it turned out. Have a good day.

 

[warri]

That was a very fast memtest - Did you do 3 passes? Preferably, even let it run overnight. Believe me, you don't want to end up using your NAS with faulty RAM.

When you are up and running again, do some stress testing before you put the NAS in use, e.g. fill up the pool with some trash data, let a scrub run and see if everything behaves correctly. After random kernel faults I'd have a hard time trusting the system again ;) And of course make sure you have backups of your data.

 

[Shdw]

That was a very fast memtest - Did you do 3 passes? Preferably, even let it run overnight. Believe me, you don't want to end up using your NAS with faulty RAM.

Only 2. Guess I'll have to put it through some more. I don't intend on filling it with data until I get some ECC ram.

 

[Shdw]

Ok a bit off the normal topic here, I hope that doesn't violate any laws on these forums. That said, can the guru's here tell me what they think of this board: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182821

I have a friend who is looking to buy a desktop and I may just sell off the stuff I have and go straight to server quality hardware. You guys have me a bit frazzled on using my current setup for the purposes of a server.

 

[warri]

Supermicro is generally a good choice. Have a look at the Hardware section, a lot of models have been discussed there already.

 

[Shdw]

Thank you very much warri. I will get right on that. I really appreciate all the help you've taken the time to give me here. Nice to see such a kind community, even for the stupid 0.O questions. ;)

Luckily my boss has been picking my brain about building him a computer, so I sold off the ram/board/and processor today. I'll be using that to get a server board, unbuffered ECC ram, and a server grade processor. Back when I started learning computing ECC ram was all I went with (it was the new big thing 15 years ago...to consumer grade builders like me), in the past years I've not even bothered to pay attention to if it's ECC or not for the desktop builds I've done.

 

[Shdw]

Supermicro is generally a good choice. Have a look at the Hardware section, a lot of models have been discussed there already.

Just a brief update and note; I sold my desktop board, ram and processor to a buddy and have an x10 board, 16 gb ecc unbuffered ram and a 3.1 ghz Xeon 1200 on the way.

This reply is to pose a question and possible solution to my 9.2 issues with that desktop system. With doing some more research I've noticed the FreeNAS docs note that 32 bit systems can be unstable for running zfs. My system was running in 32 bit. Still worked fine in 32 bit 9.1.1, but not in 9.2 was no bueno. Perhaps this was part of the issue, I don't know and don't have the hardware to test it anymore. Thought you folks might want to file it in the memory bank though for the next noobie like me you come across in the next ten min.

Good day all. Thank you again for the support. I don't care what dell advertises, they don't hold a candle to this team!!!

 

[cyberjock]

Well, if you are running with >4GB of RAM you'd be either stupid or had a brainfart to use the 32-bit version. The 32-bit version limits you to 4GB of RAM(just like every other OS out there). So using ZFS with 32-bit is just crazy. There's a reason why the 32-bit version is being abandoned in the near future. :)

At a certain point we have to expect people to not do certain things. If you are starting off with FreeNAS by installing the wrong version you're going to have MUCH bigger problems later. It's like someone asking "why doesn't my car start?" and you actually HAVE to ask the question "is there gas in the gas tank?" You just assume people really aren't that incompetent.

No offense...