FreeNAS 9.3 crashing, weird stuff happening (with video)

[CrunchyMetal]

Hey guys,

First off I apologize as I tried searching around and tried what I could but I'm stumped. I am also not sure where I can find any logs or anything to help me narrow down exactly what's happening at the time of failure.

The video is here: https://www.youtube.com/watch?v=78ueJXbR0EM

So nothing has really changed lately that I can think that would cause this, but the problem seems to be getting progressively worse. The system seems to be crashing, then rebooting itself and hanging on the boot. It will freeze somewhere on the GRUB loading portion (forgive me as I am a linux/freebsd noob, not sure entirely what this is). It has also frozen right in its tracks, and pressing the power button won't get it to shut down, I have to hard reset it, though its typically the first scenario where it reboots itself.

It started happening maybe a month ago, once randomly and I thought nothing of it. Just rebooted the system and it was fine. Then it started happening every Sunday around the same time. I started thinking it was related to the long SMART tests I had scheduled for every Sunday (though not at that time, they were scheduled for 2AM, kept freezing around 7-8 pm). I have since deleted all these scheduled tests and same thing.

Now it happens very frequently, especially if I try to do anything with FreeNAS. Anything more than one or two changes in the webGUI and it will crash, reboot itself and hang on the boot. Its also becoming more difficult to get it to come back up after it reboots itself. It takes several hard resets to get it past the GRUB. Recently I tried watching the console as it crashed and saw some really strange stuff flashing on the screen, and was able to record a few seconds of it.

You can see it crashing then rebooting and freezing on the GRUB.

I read elsewhere that hanging on GRUB could be a result of a bad flash drive (which I was booting from a 4GB) and that the new recommended size of boot drive is 8GB, so I reinstalled freeNAS onto a 16GB and reloaded the config. Same problem. I also tested the RAM, 6 passes through memtest came out clean.

Relevant hardware:

Corsair CX500 PSU
8GB DDR3 RAM
ASRock A75M-ITX FM1 AMD A75 (Hudson D3) Motherboard
AMD A6-3500 Llano Triple-Core 2.1GHz
5x WD 2TB Greens

Any help would be greatly appreciated. Thanks in advance!

 

[SweetAndLow]

I would start by reinstalling the os on a new USB stick. Make sure to save and upload your configuration file. I would also run a memtest. Then finally if it's still acting strange replace the power supply.

 

[CrunchyMetal]

Well I already did those first two, so you think its the PSU? Is it possible its something in the config itself? I didn't think it could be since its hanging on the boot, but like I said I'm a bit of a noob so I'm not entirely sure what's possible.

 

[SweetAndLow]

Well did it boot correctly when you reinstalled the os? That would have made it clear that it's the configuration file.

Your hardware isn't what we call highly recommended and could have compatibility issues. Does freenas have any warning messages?

 

[CrunchyMetal]

It did boot correctly, though I honestly didn't leave it at stock long enough to know for sure if it would freeze, as sometimes it can go a day or two without crashing. I just booted it up once and loaded the config.

No FreeNAS warnings either.

 

[CrunchyMetal]

Thinking maybe I should restart back to vanilla and see if it keeps doing it?

 

[CrunchyMetal]

Ok so I wiped the flash drive and did a fresh install. It seems to be working fine and have not experienced any crashes yet but now I'm getting this FreeNAS warning message: "The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected."

 

[SweetAndLow]

Ok so I wiped the flash drive and did a fresh install. It seems to be working fine and have not experienced any crashes yet but now I'm getting this FreeNAS warning message: "The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected."

That means that you had some data corruption and ZFS fixed it for you. Nothing to worry about since you didn't get an email about smart tests failing or anything else going wrong on your system. It was probably just from all the random reboots you have had.

 

[CrunchyMetal]

Ah alright. Well then I guess I'll just keep an eye on it and start rebuilding from scratch and see if it freezes up anymore. Thanks a bunch.

 

[CrunchyMetal]

Nope it crashed and rebooted, hanging on the GRUB menu again. Is there any way to figure out exactly what's happening at the time of the crash? Any log or dump files I can lookup? Am I correct in thinking that this is more likely a hardware issue now?

 

[CrunchyMetal]

So after a couple weeks of no crashes I wanted to come back and post my resolution in case anyone else has this problem. The USB ports on the back panel were bad. After moving the flash drive with FreeNAS on it to a USB port on the front of the case there have been absolutely no issues, had about 2 weeks solid uptime.