FreeNAS disappears like it went to sleep

[ghamauricio]

Good morning all. This is my first post here.

So I bought a Lenovo Thinkserver TS150, basic config (8GB RAM), running two 8TB mirrored HDs. Current version (latest stable).

The thing is, after a while without use (sometimes at the end of the day, sometimes early in the morning) my FreeNAS box will look like it went to sleep.

No computer will be able to access the shares, I can't ssh into it, and sometimes my Mikrotik router will show in the "DNS Server" page that there is no IP assigned for it, like it's offline. Still, funnily enough, it will reply to pings.

If I try to connect a monitor and keyboard to it, nothing shows on the screen. Even if I type something.

All I could come up with was a hard power off.

How can I start to troubleshoot this?

Thank you for the help!

Best regards,
Guilherme Mauricio

 

[SweetAndLow]

Is it on? Are the fans spinning, light blinking or hdds spinning? Check /var/log/messages for more info around the time it goes down.

 

[Chris Moore]

I would suggest going into the BIOS / UEFI firmware settings and looking to see if there are any power management settings that you can disable. I wouldn't expect it on a server, but many new computer ship with power management features enabled in the BIOS. It can't hurt to look.

 

[ghamauricio]

Ok, so this time, upon plugging a monitor, console shows up. But it's hard to read (4k resolution, last couple of lines are below what the monitor is printing).

Here is a screenshot of the console (the blank lines were me hitting enter trying to bring the text up to read the result. Last command was ctrl+alt+esc and now nothing more happens).

Still inaccessible through network, although the router indicates it's got the up assigned).

 

[ghamauricio]

Ok so I decided to run memtest, forced multitasking, and now then it got stuck in this:

Won't answer to keyboard, and keyboard lights are unresponsive (caps lock was on, and it remains on no matter what. Num and scroll lock are off and so they remain).

Hard power off.

What now? :(

 

[ghamauricio]

Allright, ignore last post. I was running the multithreaded version of the test, which by the internet's accounts, is buggy.
I've succesfully run the single thread test again and all is going well.

 

[ghamauricio]

Still struggling here.

First, sometimes it looks like I can't use the CLI.
If there is some message (ex.: link state is down / link state is up), I can't use the keyboard attached to the machine anymore.

Second, the IP for the NAS server is reachable by ping, but sometimes the router will not show it on the list of connected devices.

Third, I can't access the web GUI nor SSH into it.

Also, power off doesn't gracefully shuts down. I kinda have to hard power it off. And after a fresh boot, it works for a few hours (don't know exactly how many).

What the hell am I doing wrong?!

 

[ghamauricio]

I would suggest going into the BIOS / UEFI firmware settings and looking to see if there are any power management settings that you can disable. I wouldn't expect it on a server, but many new computer ship with power management features enabled in the BIOS. It can't hurt to look.

There aren't many options in the BIOS, only thermal/accoustic performance.

 

[Redcoat]

I'd start with a fresh install of FreeNAS on another USB stick in order to (attempt to) eliminate the possibility of a corrupted boot drive.

Then I'd look to see if the CPU is overtemping - could it have a malfunctioning cooler or is the thermal connection between the cooler and the chip poor?

Finding this sort of thing can be painful ...

 

[ghamauricio]

There aren't many options in the BIOS, only thermal/accoustic performance.

Actually, I double checked and there was an option for extended power saving something.

Anyway, I added S5 funcionality to the power button. At least now I have a way to cleanly shut down the server in case everything goes havok again.

So I went to the legacy web GUI and bound the IP to the web GUI server. Weird it's not automatically done.
Also, when I went to interfaces, there were none. I added it, and added a fixed IP alias just in case.

By the way, the logs report some weird stuff about the usb key being removed (although I can swear it wasn't), like this:

Code:

Mar 22 00:32:01 freenas ugen0.3: <Lexar USB Flash Drive> at usbus0 (disconnected)
Mar 22 00:32:01 freenas umass0: at uhub0, port 19, addr 2 (disconnected)
Mar 22 00:32:01 freenas da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
Mar 22 00:32:01 freenas da0: <Lexar USB Flash Drive 1100> s/n AARMRUG6I3O6XZ96 detached
Mar 22 00:32:01 freenas (da0:umass-sim0:0:0:0): Periph destroyed
Mar 22 00:32:01 freenas umass0: detached

I'll get new flash drives. In the meantime, here is the whole log attached.

 

[bphilpott]

I've had the same issue as you. I can ping, but nothing else. SSH tries to work - it prompts for a password, but nothing after that, CIFS does not work, and even using a keyboard/monitor, it is largely unusable. It will attempt to log in but just hangs. And it is frustrating because I have to drive an hour to shut it down and bring it back up to diagnose.... It works fine for a few hours, but when the nightly backups try to run, they fail because the FreeNAS is unresponsive (I have the 8 bay mini).

The only thing different from yours is that when I hard power off then I have to reset the network config...

I am suspecting a memory issue at this point. I am going to look in the logs later today if I can see anything that might suggest that or maybe as you've seen that I might have a USB stick issue. I ran badblocks on my memory stick and it came back clean.

 

[ghamauricio]

Ok so the new pen drive seems to have solved the issues. Weird, since I've read countless times that FreeNAS runs off of the RAM once it's loaded, and that if you don't want to save any new settings, you could even remove the usb stick after booting. Doesn't seem to be true, even though I wasn't testing this.

There's only a couple of days worth of experiment with this new usb stick. Thanks for all the suggestions, guys! Will keep observing it.

I've had the same issue as you. I can ping, but nothing else. SSH tries to work - it prompts for a password, but nothing after that, CIFS does not work, and even using a keyboard/monitor, it is largely unusable. It will attempt to log in but just hangs. And it is frustrating because I have to drive an hour to shut it down and bring it back up to diagnose.... It works fine for a few hours, but when the nightly backups try to run, they fail because the FreeNAS is unresponsive (I have the 8 bay mini).

The only thing different from yours is that when I hard power off then I have to reset the network config...

I am suspecting a memory issue at this point. I am going to look in the logs later today if I can see anything that might suggest that or maybe as you've seen that I might have a USB stick issue. I ran badblocks on my memory stick and it came back clean.

I would test your memories first and foremost.
Also, if I had to drive to turn the thing off, I would virtualize the hell out of it and remotely shut it down, even if that doesn't solve your problems with the NAS behaving weirdly.

I wish you good luck, bro!

 

[Redcoat]

Weird, since I've read countless times that FreeNAS runs off of the RAM once it's loaded, and that if you don't want to save any new settings, you could even remove the usb stick after booting. Doesn't seem to be true, even though I wasn't testing this.

Used to be so, but not for some long time now (since intro of FreeNAS 9.3 at the end of 2016, I think).

Glad you are up again. Many of us here have moved to small (cheap) SATA SSD's (or old laptop hard drives) for boot drives because of the lack of reliability/short life of USB sticks.

 

[ghamauricio]

The problem with going the SSD route is that I wanted to be able to pass the whole AHCI controller to the guest in case I wanted to move FreeNAS to a guest VM under ESXi, but it turns out it isn't possible with the ThinkServer TS150: the controller doesn't show up in the virtualizable interfaces anyway. Plus, ESXi doesn't recognize USB storage as usable for it's working, so I might reconsider going SSD for the OS now...

 

[bphilpott]

I ran mem test on mine and the memory came back clean, so I decided to back up my config and reinstall to a new boot drive. It's only been one day, but that is longer than it normally goes before becoming unavailable. I previously had a SATA flash drive that comes in the XL-mini, but I replaced it with a laptop hard drive that I had sitting around and the chassis had a place to mount a 2.5" drive