Is there a better way to poll my UPS for self-test status?

[keboose]

My system is running on an APC Back-UPS RS1300 LCD battery backup. It's not the newest piece of hardware, but I can still connect via USB, and I have the UPS service set up to shut down the server if the battery-remaining time falls below 2 minutes.

The problem is that I recently had to replace the batteries, because one failed. It straight up would not hold a charge, and that was when I realized that the UPS doesn't seem to perform automatic self tests aside from initial power-on. At the least, there is no indication that a test has been run by polling the UPS via the shell. Looking through the manual on the APC site only describes how to manually initiate a self-test via the front panel, but I want to do that automatically.

Before I go and put the work in, I just want to make sure: is there already a way to check the battery status of the UPS that is baked into the service? If not, I would have to set up a cron job to do it for me.

Below is all the status flags I can pull from the UPS. There are no variables for test interval or date of last passed test like many other UPS's.

Code:

battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2010/00/39
battery.mfr.date: 2008/08/10
battery.runtime: 4188
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 26.7
battery.voltage.nominal: 24.0
device.mfr: American Power Conversion
device.model: Back-UPS RS 1300 LCD
device.serial: [redacted] 
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: /dev/ugen0.3
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 139
input.transfer.low: 88
input.transfer.reason: input voltage out of range
input.voltage: 121.0
input.voltage.nominal: 120
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.firmware: 838.H7 .D
ups.firmware.aux: H7
ups.load: 9
ups.mfr: American Power Conversion
ups.mfr.date: 2008/08/10
ups.model: Back-UPS RS 1300 LCD
ups.productid: 0002
ups.realpower.nominal: 780
ups.serial: [redacted]
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

If there is no functionality in the system already, this would be my plan of attack.
First, I would initiate a test by running the command:

Code:

upscmd upsnam@localhost test.battery.start.quick

and then continually poll the ups with the command:

Code:

upsc upsnam@localhost ups.test.result

Looking for a result "Done and passed". A slight hangup is that the test result variable resets to "No test initiated" after a few minutes, which explains why I can't tell if this thing actually does automatic tests. At any rate, I would have this script run every few days and email me if the test result differs from either "Done and passed" or "no test initiated". I would probably also want to run the "test.battery.start.deep" test every month or so, as well.

So am I crazy, and there is already something in the UPS service that does this? Or does this sound like a good idea?

 

[jgreco]

The Back-UPS series of UPS units has typically been considered to be a consumer-grade unit, which means that it's unlikely to sport features such as fancy automatic self-testing. For scheduled self-testing, you need a Smart-UPS.

 

[keboose]

Ah. I see. I was fooled by the APC page that said it was "High Performance Battery Backup & Protection for Business Computers". Not a dig at you; It's probably meant for workstations, considering it came with the Windows software on a disk.

Do you think the ups.test.result variable is a good indicator that I can use to monitor the battery state? I would prefer writing a bash script over buying a new UPS.

 

[jgreco]

Ah. I see. I was fooled by the APC page that said it was "High Performance Battery Backup & Protection for Business Computers". Not a dig at you; It's probably meant for workstations, considering it came with the Windows software on a disk.

Not considered a dig; I absolutely hear your frustration. This is product differentiation at its worst. The vast majority of the cost of a UPS are the chassis, inverter and charging electronics, and the batteries. The processor that powers these things is a ten cent component, and while there might be a slightly better one for the Smart-UPS... well let's just say that I glanced right now and a Back-UPS 1500 is around $180 while a Smart-UPS 1500 is around $350 - almost twice as much. You might get better parts with the better UPS, which may justify some of the extra cost. However, what we're talking about in this thread is omitting features in order to drive people towards the Smart-UPS for features.

Do you think the ups.test.result variable is a good indicator that I can use to monitor the battery state? I would prefer writing a bash script over buying a new UPS.

Couldn't tell ya, sorry.

 

[keboose]

True, gotta love companies putting needless software locks on their products.

I wrote a script in Python (I'm no programmer, so it's easy for me) to check the battery once a week. It runs fine from the shell, and I made sure all of the exit conditions work, but when I set it up in a cron job and run it, I don't get any emails for standard or error output (I unchecked both 'hide output' boxes on the job, and I get pool scrup emails fine, so email is set up correctly). Would you mind having a look and telling me what I did wrong?

Code:

#! /usr/local/bin/python

import os
import sys
from time import sleep

print('Running UPS quick battery test...')

os.popen('upscmd -u admin -p [password] ups@localhost test.battery.start.quick')
sleep(5)
result = os.popen('upsc ups@localhost ups.test.result').read().rstrip()

i = 0
while i<20:
        if result == 'Done and passed':
                print('complete!')
                sys.stderr.write('complete!')
                sys.exit(0)
        elif  result != 'No test initiated':
                print ('UPS BATTERY TEST ERROR! BATTERY TEST FAILED:\nups.test.result: "'+result+'"')
                sys.stderr.write('UPS BATTERY TEST ERROR! BATTERY TEST FAILED:\nups.test.result: "'+result+'"')
                sys.exit(1)
        i+=1
        sleep(5)
        result = os.popen('upsc ups@localhost ups.test.result').read().rstrip()

print ('Timeout! UPS did not respond in a resonable time for self-test!')
sys.stderr.write('Timeout! UPS did not respond in a resonable time for self-test!')
sys.exit(2)

 

[jgreco]

I don't really do Python as I'm an old school C and shell script kinda guy. I'd only try writing the code if I had the gear here to test it.

 

[droeders]

It runs fine from the shell, and I made sure all of he exit conditions work, but when I set it up in a cron job and run it, I don't get any emails for standard or error output (I unchecked both 'hide output' boxes on the job, and I get pool scrup emails fine, so email is set up correctly).

If you're certain it runs fine from the shell but not from cron, I'd suspect a PATH issue first.

Can you change your script to use the full path to anything you execute with os.popen() (i.e. /path/to/upsc instead of upsc)?

 

[keboose]

Sorry, I wasn't being clear: the script does work fine when run from cron, except I do not get emails, even though the script prints to stderr. At the moment, I have gotten around that by replacing the 'stderr.write' lines with:

Code:

os.popen('echo "Quick Test result: ' + result + '" | /usr/bin/mail -s "UPS WEEKLY TEST: [PASSED/FAILD/TIMEOUT]" [my.email@addr.ess]')

(where 'result' is the variable in the script from my last post) to explicitly email me the result, rather than relying on the cron reporting system.

This is working (so far) from cron. The only variable here is that I don't actually know what this will do when a battery fails, since the UPS is working fine at the moment.

I also found out there is a more intense test that can be run by replacing test.battery.start.quick with test.battery.start.deep. This will switch the UPS over to battery power until battery level reaches 25%. It's recommended to be run no more than every 3 months, and I plan to do that, though I would also probably have to run some kind of stress test at the same time, otherwise it would take hours to drain (UPS estimates 4,800 minutes battery time when server is idle.)

 

[droeders]

I see this in the 11.2-U3 documentation for the stdout/stderr options: "When unset, any standard output is mailed to the user account cron used to run the command."

I assume your cron job is running as root?

Is your root user account setup with an external email address? For example, see the first note in this section:

https://www.ixsystems.com/documentation/freenas/11.2/accounts.html#users

 

[keboose]

Yes, my root account has my email address in the account settings. I get pool scrub and other system warnings with no issues.

I do have the same email set in my normal user account, as well. Could that have an effect?

 

[droeders]

If the cron job runs as the root user and the root user email is set, I think that should be all you need for cron.

What happens if you run this on your FreeNAS box?

echo "hello world" | /usr/bin/mail -s "Testing" root@localhost

Do you receive this testing email at your external address?

 

[keboose]

The cron job kicked off last night and, drumroll... I got both the script email AND the cron report of stdout!

So that means I DON'T get reports when I run cron jobs manually from the gui? That is all that differs from when I was testing...

To answer your question: no, I don't get emails when running that command, either as my user account, or as root

 

[jgreco]

File a bug report if you're not getting the stdout when you run cron jobs manually from the GUI.

 

[keboose]

Created a bug report: https://jira.ixsystems.com/browse/NAS-101569

 

[keboose]

I see the report has all but been completed. In the meantime, this is what I am using for running regular ups tests via cron:
ups.quick.py

Code:

#! /usr/local/bin/python

import os
import sys
from time import sleep

def sendMail(result = "", errFlag = 0):
    result = '"Quick Test: '+result+'"'
    emailAddr = '[my@email.address]'
    if errFlag == 0:
        os.popen('echo '+result+' | /usr/bin/mail -s "UPS WEEKLY TEST: PASSED" '+emailAddr)
    elif errFlag == 1:
        os.popen('echo '+result+' | /usr/bin/mail -s "UPS WEEKLY TEST --- FAILED! ---" '+emailAddr)
    else:
        os.popen('echo "Operation timed out! UPS did not pass/fail in the alloted time frame." | /usr/bin/mail -s "UPS WEEKLY TEST --- ERROR! ---" '+emailAddr)
       
os.popen('upscmd -u admin -p doitnow ups@localhost test.battery.start.quick')

i = 0

while i<20:
    sleep(5)
    test = os.popen('upsc ups@localhost ups.test.result')
    result = test.read().rstrip()
    test.close()
    if result == 'Done and passed':
        sendMail(result,0)
        sys.exit(0)
    elif  result != 'No test initiated':
        sendMail(result,1)
        sys.exit(0)

    i+=1

sendMail('',2)
sys.exit(0)

It is watching for strings that may be unique to APC battery backups, so it may not work with other systems. I also set up a 6-month deep cycle test (with the Cron Last Sunday script as a helper, it runs the first saturday of January and June so as not to conflict with my every-friday quick tests), which is a mostly identical script. The differences are: it sends the command test.battery.start.deep, I set the sleep duration to 10(seconds), and the while loop to 720 iterations (for a total of 2 hours, which should be long enough considering the idle load of my NAS.)