SOLVED Shutdown and reboot problems

[74m]

Hey guys,

my FreeNAS server (specs below) wont shutdown. More precisely: If i do a "shutdown -h now" via ssh, the shutdown process will stop (picture below), so that i have to do a hard reset.



If i try a shutdown via the GUI, my FreeNAS wont shutdown, instead it does a restart.

I guess its the same problem, but the shutdown process via GUI will prevent the halt during the shutdown, so it will forces a restart.

So what can i do now?

Thank you... :)

Greetings
74m

Edit: Ok, i found out that the right command ist "shutdown -p now", after this, my system will reboot - just like i click the shutdown in the GUI... ;(

 

[Sakuru]

Do you have any "Turn on after power loss" type features enabled in the BIOS?

 

[74m]

This function is disabled in my bios.

-I also tried to unplug the UPS. I thought maybe the UPS will power on my server after a shutdown via usb... - no change!
- So i tried to unplug all power, LED and reset connectors on the board - no change!
- If i try a "init 0" via SSH, the system will reboot, just like shutdown -h/P and the shutdown via GUI.
- If I do a "halt" via SSH, the system will do all the steps for the shutdown and "halts". Then i press my power button for a few seconds and my server will go down - and STAY DOWN!
- A "Power off" via IPMI results in a power down (;)) - and STAY DOWN!

The strange thing is: The system really shut down. Even the fans will stop. And then, after 2 seconds, my system will boot up again.

 

[joeschmuck]

Glad you solved it. It does matter what switches you put into your command line.

 

[74m]

Hey Joe,

Ive not solved it! I'm still not able to shut down my freenas via GUI or CLI, so that it stay down. Physical interaction or IPMI cant be the solution...

Greetings
74m

 

[joeschmuck]

So let me understand this...

You can power down the computer but after it powers down, it starts back up fairly quickly?

If this is true, command the system to power down and once it start to power off, unplug all your LAN connections to your FreeNAS box. Does it stay powered down now? If yes, you need to go into the BIOS and see if there is a setting called Wake-On-LAN and disable it, or it will be something similar. I don't have a Supermicro MB but they are all similar. I'm not sure if the IPMI could be causing it either.

This is the only thing I can think of off the top of my head.

 

[74m]

Heyho,

joe, you got it. This is my problem! ;)

so, after a bios update the system will stay down SOMETIMES!
I cant find any reproducible system behind this behaviour...

But it is not wake-on-lan, because my onboard ethernet ports dont support WOL. I'm sure about this, because i bought a new intel ethernet PCIe card just for this feature... I've installed the new card today, but the shutdown/reboot-thing was already before i had the new card.

 

[joeschmuck]

Please clarify what you mean by this is your problem. I need you to be very descriptive so I'm not second guessing if we are talking the same thing. This will speed things up.

So is it working 100% correctly, meaning the power turns off and remains off if you remove the LAN connections? Also, how many LAN cables do you have connected? re you using IPMI too?

Lets say it's not the LAN connections, have you disabled the Backplane USB Wake_Up Jumper to Disable (Enabled by default)? This could be causing the issue as well.

 

[74m]

Sorry Joe, maybe my english is too bad...
Problem: I shutdown the freenas, and i will reboot!

I've done 8 tests, 4 with LAN and IPMI plugged in and 4 without LAN and IPMI plugged in. Now, in all tests the server stayed down. I have to do more testing. Actually i think, the reboot will happen when i have the ipmi (kvm window) open... Because in all the last 8 tests, i did not had the ipmi window open!

 

[74m]

Okay, there a news about my shutdown problem:

Since Dezember 2015 i used a simple shell script for shutdown. It checks 2 IPs in my LAN, if these IPs are not reachable, the server will shut down. If one or both IPs reachable (ping) the server stays online. I've added a cronjob wich starts the script every 15 minutes in a given timerange. So far, so good!

I figured out that my server always need 2 attempts for a shutdown. I've got an email when the script passed through successfully. Every evening i got 2 Mails. After the first shutdown my server reboots immediately. After the second email (= the second shutdown, approximately 15 minutes after the first) the server will stay down.

This is 100% reproducible every evening! So I think there are some programs (daemons or similar) that are loaded after a while of using my freenas, wich will affect a reboot instead of a shutdown. Maybe these programs are not started after the first reboot, so that the second shutdown works after 15 minutes on-time.

Why does every second shutdown work?

Greetings
74m

 

[joeschmuck]

Can you post your script please, it may help figure it out. Off the cuff I'd say you have a variable issue.

 

[74m]

Sure, here it is. Very simple and static...

Code:

#!/bin/bash

_exit () {
case $1 in
1) echo "no shutdown" ;;
2) echo "shutdown in 180 sec" ; shutdown -p now ;;
esac
exit $1;
}

# check if my hardware is online
# 123 = iMac
# 124 = TV
if [ `ping -c 5 -i 1 192.168.0.123 | grep -wc 100.0%` -eq 0 ] ; then _exit 1 ; fi
if [ `ping -c 5 -i 1 192.168.0.124 | grep -wc 100.0%` -eq 0 ] ; then _exit 1 ; fi

# if all hardware is offline, wait for 3 mins
echo "all hardware is offline - shutdown in 3 mins"
sleep 180

# second check
if [ `ping -c 5 -i 1 192.168.0.123 | grep -wc 100.0%` -eq 0 ] ; then _exit 1 ; fi
if [ `ping -c 5 -i 1 192.168.0.124 | grep -wc 100.0%` -eq 0 ] ; then _exit 1 ; fi

# shutdown ...
_exit 2

 

[joeschmuck]

So you run this script every 15 minutes from CRON in FreeNAS and during the shutdown/reboot FreeNAS sends you an email that the system is being shutdown? Do the emails always say shutdown or for the reboots, is the email a reboot message?

 

[joeschmuck]

Additionally, does this behavior only happen when CRON runs the script or does it also happen if you manually run the script? Meaning that you have to run the script "sh script.sh" and the system reboots, then you get into the shell again and type "sh script.sh" and then it shuts down. This would help to know that.

 

[joeschmuck]

Lastly, what does your cron entry look like? You are not running this from another script are you in order to get your email notifications?

Right now your script works perfectly for me from the command line. I'll wait for a response and then create a CRON job to see what happens.

I placed your script in the CRON and it works perfectly as well.

 

[74m]

Hey Joe,

thank you for your help!

1)
I've set up the email notification in the basic freenas settings. So i get a email for every cron job (eg scrub). So i dont get a detailed response, just a notification that the cronjob was executed. Looks like this:

Code:

Subject:
Cron <root@FreeNAS> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" sh /mnt/Jails/scripts/shutdown.sh > /dev/null

Mail:
shutdown: [pid 33466]

2)
There is no difference whether i start the script manually or via a cronjob (or a simple shutdown command via ssh...). It seems that if the freenas booted up a very short time ago, the shutdown will work. But if I wait to long and the freenas server was in use, any shutdown will turn into a reboot. I just want to know what services / deamons / programms are responsible for this behavior.

3)
There is just one other cronjob: A backup download from a online server (via scp), executed once a day... Im pretty sure there is no connection to my shutdown problem!

 

[joeschmuck]

Beats me to be honest with you. Your script works fine for me but then again I don't have the same hardware you do either.

I think the only other thing you can do is see what your log files say. I'd be curious what the difference is between a shutdown and reboot. There must be something in there that is different, either that or there is something odd happening specifically with your motherboard.

Hardware things you could try is to remove all add-on cards and try to command the shutdown via the console.
Replace the power supply in case it's not turning off properly for some reason.

Have you ensured that both the BIOS and IPMI firmwares are the most current? I thought we did address that earlier.

 

[74m]

Thanks joe!

I could try a BIOS and IPMI upgrade, but i've already done that for about 6 months... So I'm not running the first stock firmware.

I guess I should try difference tasks to see after which service my freenas will reboot instead of a shutdown. What logfiles should i check? /var/log/messages?
I dont even know if the log overcome multiple boots. I'll check this and report later! ;)

 

[joeschmuck]

I am assuming you are running the cron job as user root.

Yes, /var/log/messages typically will contain a few days worth of data so you can look for the date/time that the reboot/shutdown occurred.

In a shell type the following:

Code:

cd /var/log
cat messages | grep shutdown

The message which should show up is:

Code:

[root@tfreenas] ~# cd  /var/log
[root@tfreenas] /var/log# cat messages | grep shutdown
Mar 25 16:16:16 freenas shutdown: power-down by root:
Mar 25 16:51:05 freenas shutdown: power-down by root:
Mar 27 18:15:21 tfreenas shutdown: power-down by root:
Mar 27 18:21:05 tfreenas shutdown: power-down by root:
Mar 27 18:28:04 tfreenas shutdown: reboot by root:
[root@tfreenas] /var/log#

Note that the first three are my normal results and the message clearly indicated "power-down". The last message indicates "reboot" but I commanded a reboot so I could see what the message would be, this wasn't an accidental reboot.

Now I'm curious what your message results will be.

 

[74m]

I got fresh messages log after each reboot. So i cant search for power-down...
???