SOLVED Script to gracefully shutdown Bhyve Windows VM during FreeNAS reboot/shutdown problem

[Chris123]

Problem: Cannot use 'curl' in 'Tasks > /Init/Shutdown Scripts': "curl: not found". I'm using curl in a script to shut down Windows VM's gracefully.

The only way I have come across to ensure gracefully shutdown a Windows 10/2016/2019 bhyve vm (created in gui) is to ssh in to the vm and issue a shutdown from there. (The bhyve 'poweroff' kills the Windows VM and the 'stop' is very inconsistent - sometimes it works and sometimes not). Then, to figure out when the VM actually has stopped (after the ssh shutdown command) I use curl to access the FreeNAS api (2.0). To my knowledge there are no bhyve command line wrapper that interface the FreeNAS bhyve gui. I've tried iohyve, vm, bhyvectl, and none of them worked for the gui-created VMs.

My questions are: Why can I not use curl in the freenas shutdown script? Are there any other ways (than accessing the freenas api) to tell when the state of a gui-created VM has switched from 'running' to 'stopped'? It would be great to have a command line bhyve wrapper that also works for bhyve gui-created vm's (similar to iocage for jails) of course...

Script:

Code:

#!/bin/sh

VMID=$1
VMIP=$2

initialstate=$( curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )

if [ "$initialstate" != "STOPPED" ]; then
   sshshutdown=$( ssh Administrator@$VMIP 'shutdown /s /t 0' 2>&1 )
   if [ `echo $sshshutdown | wc -w ` -ne 0 ]; then
      printf "Shutdown failed VM $VMID: $sshshutdown"
      exit 1
   fi
fi

state=$( curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )

counter=0
while [ "$state" != "STOPPED" ]; do
  state=$( curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )
  counter=$((counter+1))
  sleep 10
  if [ `echo $counter` -gt 1080 ]; then
    printf "Shutdown failed VM $VMID: \n$state"
    exit 1
  fi
done

There was a ticket on bhyve vm shutdown some time back, but whatever was done doesn't seem to work for Windows 10/2016/2019. I do however understand that without guest tools installed in the VM shutting down gracefully can be tricky.

 

[colmconn]

Have you tried using its full path (/usr/local/bin/curl) instead of relying on the PATH environment variable?

 

[Chris123]

Have you tried using its full path (/usr/local/bin/curl) instead of relying on the PATH environment variable?

That worked of course!! Silly of me not thinking about that. I though maybe 'curl' weren't allowed during shutdown for whatever reason.

Thanks for your help!

Code:

#!/bin/sh

VMID=$1
VMIP=$2

initialstate=$( /usr/local/bin/curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )

if [ "$initialstate" != "STOPPED" ]; then
   sshshutdown=$( ssh Administrator@$VMIP 'shutdown /s /t 0' 2>&1 )
   if [ `echo $sshshutdown | wc -w ` -ne 0 ]; then
      printf "Shutdown failed VM $VMID: $sshshutdown"
      exit 1
   fi
fi

state=$( /usr/local/bin/curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )

counter=0
while [ "$state" != "STOPPED" ]; do
  state=$( /usr/local/bin/curl --silent --show-error -u root:<password> "http://192.168.1.14/api/v2.0/vm/id/$VMID" | grep "state" | awk '{print $2}' | tr -d ',"' )
  counter=$((counter+1))
  sleep 10
  if [ `echo $counter` -gt 1080 ]; then
    printf "Shutdown failed VM $VMID: \n$state"
    exit 1
  fi
done

 

[michaeleino]

I'd rewritten the script with the same logic of yours, but using the API not SSH

Code:

#!/bin/sh

fnhost=localhost
vmAPI="http://$fnhost/api/v2.0/vm"
username=root
password=PASSWORD
backoff=3 ##wait n seconds between hitting the next VM stop command
safe_hold=10 ## wait n seconds at the end of initiating ALL shutdowns to make sure ALL VMs is off.
safe_cnt=20 ## how many times we will repeat the safe_hold above
curl=/usr/local/bin/curl ## define curl location to be able to execute as a task

exec > /dev/console
exec 2>&1
get_running()
{
  echo "Getting running VM Names/IDs"
  RunningIDs=$($curl --silent -u $username:$password -X GET "$vmAPI" | jq '.[] | select(.status.state == "RUNNING") | .id')
  Runningnames=$($curl --silent -u $username:$password -X GET "$vmAPI" | jq '.[] | select(.status.state == "RUNNING") | .name')
}

check_running_then_shutoff()
{
  get_running
  if [ -n "$RunningIDs" ];then ## check if IDs is not empty
    echo "$Runningnames is Running";
    for VMID in $RunningIDs
    do
        shutdown_response=$($curl --silent -u $username:$password -X POST "$vmAPI/id/$VMID/stop")
          if [ $shutdown_response == "true" ]; then
            echo "Sent Stop signal to VMID = $VMID"
          else
            echo "Failed to send API request to shutdown VMID = $VMID , API response = $shutdown_response"
          fi
        echo "Sleeping for $backoff backoff seconds..."
        sleep $backoff
    done
    echo "Sleeping for $safe_hold safe hold seconds..."
    sleep $safe_hold
    still_running #check and wait more
  else
    echo "No running VMs"
  fi
}

still_running(){
  counter=0
  echo "Will check $safe_cnt times every $safe_hold seconds..."
  while [ $counter -le $safe_cnt ]; do
    counter=$((counter+1))
    echo "Check $counter/$safe_cnt"
    get_running
    if [ -n "$RunningIDs" ]; then
      echo "$Runningnames is Still Running, still waiting..."
      echo "Sleeping for more $safe_hold safe hold seconds..."
      sleep $safe_hold
    else
      break
    fi
  done
  echo "Exiting, no more running VMs or VMs not going to shutdown softly."
}
## Running functions
check_running_then_shutoff

you need to set the password of your root account, adjust the timing counters as your needs, and set the script in shutdown tasks, noting that you need to assign task timeout larger than the total values assigned in the vars section.

Hope FreeNAS integrate something with this functionality.. as it is mandatory for any hypervisor!

Thanks all