system dies periodically

[richardmh]

Dear all

This is a HP microserver:
FreeNAS Build FreeNAS-8.0.4-RELEASE-p3-x64 (11703)
Platform AMD Turion(tm) II Neo N40L Dual-Core Processor
Memory 8049MB
OS Version FreeBSD 8.2-RELEASE-p9
4 x 1tb with one ZFS raid 2 volume, various CIFs shares and one isci on top of that.
Active directory, cifs, smart, SSH and ISCI services enabled.

All runs perfectly for a couple of weeks and then quite quickly the whole thing becomes unresponsive, no connections available, no web ui. I do sometimes seem to still be able to get in to the console via SSH (more below)

Basically my current fix is to pull the plug on the box and it reboots back to normal, but this is obviously not a good thing.

SMART on one of the disks is reporting Device: /dev/ada3, 2 Offline uncorrectable sectors but this seems to be stable so I haven't bothered to replace the disk yet, though I do have one if it gets worse.

Questions:
1) At least a soft reboot via SSH would be better - but what command do I use?
2) Might the problem be logged somewhere? Where would I find this?
3) Setting up a scheduled reboot every week at some convenient time might be a solution, albeit not very satisfactory.... how?
4) Is there some other setting I could adjust which might fix the problem more permanently?

Thanks in advance

Richard

 

[cyberjock]

1. "shutdown -r now' will reboot the server. 'shutdown -p now' will shutdown the server.
2. The logs are in /var/logs. Do note that if you reboot the logs will be wiped so you will lose any evidence of what's going on.
3. You could create a cron job to execute the command in #1. Consult the manual on creating a cron job.
4. I don't know of any off the top of my head. Maybe someone can provide better info for that.

What does your swap space usage look like? If you see it steadily going up you may need more RAM in your system.

 

[richardmh]

1) Thankyou for the commands, I will try it.

2) I'm not very good at command line stuff in linux, can you point me to a simple guide to navigating getting to /var/logs and then opening them?

3) I'll try it, thankyou

Swap space doesn't seem to be anything untoward, over the last month is showing free: 8.5G min 8.8G max. In any case 8 gig is the max you can put in a hp microserver and there seems to be a lot of consensus out there that this is a really good budget box for freeenas.

Looking at the graphs in system --> reporting there doesn't seen to be anything untoward at all, activity seems to be more-or-less the same every day (busiest in the night when backups are being done), and then (in those graphs which maintain their record) everything just stops until I reboot - is a bit like something is completely crashing.

Thanks

Richard

 

[cyberjock]

2).At the console type 'cd /var/logs', then type 'ls' for a list of files in the directory. You can open them with either 'vi logname' or 'ee logname'. If you aren't familiar with vi I'd use ee. Hit ESC to get the menu. ;)

Note that file names are case specific.

 

[richardmh]

OK, this is very helpful, thankyou.

I've had a poke around, can't find anything unusual, but then, as you say, the logs are wiped since the last time it crashed and I pulled the plug.

I suppose if I can get in by SSH when freenas has stopped working means freeBSD is still working while the app freenas has crashed. This suggests there must be some command which I can use to restart freenas without having to reboot the entire box.

Any idea what it is?

Of course I might be misunderstanding what freenas is, but if it is just an app on freeBSD then I suppose there might be some fairly frequent cron command I can set up to restart freenas if it stops working..... freenas responding - do nothing / freenas dead - restart it. Much better than just forcing the box to reboot at some arbtitrary interval I set.

Thanks

Richard

 

[cyberjock]

One of the FreeNAS admins will have to talk about any commands related to restarting FreeNAS if that's even possible. But I'm thinking your much better off trying to reboot than trying to restart FreeNAS. FreeNAS touches so many other programs that if one of them is the problem restarting FreeNAS wouldn't do you a bit of good. It will just freeze up like it is already.

You could setup a cronjob to reboot the server every night at like 1am. But to be honest, your trying to cover up the problem instead of fix it, which could backfire. You really need to figure out what is wrong and fix it. I know.. my opinion is probably obvious but you never know what this issue could manifest into next. It would suck if you lost your zpool because some piece of hardware is starting to fail and you don't figure it out until it's too late.

Do you have a backup of your data? If not I would consider trying to backup the important stuff.

 

[ben]

FreeNAS is not a single app, FreeNAS is every single program - python, a webserver, dozens of daemons - running on a small FreeBSD instance. FreeNAS is the sum of all those parts. Just reboot the box if that's what you're going for.

As for logging, there is a FreeBSD service that allows you to remotely gather log messages. See http://doc.freenas.org/index.php/Settings and http://www.freebsd.org/doc/handbook/network-syslogd.html