SOLVED UPS Mysteriously Powers Itself Off

[joeschmuck]

I have an odd problem and I'm definitely not the UPS expert.

The issue: Every few days, I'm not sure what the periodicity is yet, it has only shut down twice so far over I think 5 days maybe, I will obtain exact date/times when I do a little more digging. The UPS is powered off, no alarm, and when I bring the system back online, it comes up without any alerts on TrueNAS nor ESXi. But none of the logs say the system is shutting down which I think is odd so my initial thought is the UPS suddenly powered off.

I am looking for a possible way I could troubleshoot this strange problem. The details of the system are:
CPS LX1500GU UPS, USB port passed through to TrueNAS VM.
ESXi 7.0 (with current updates)
All of my system description is in the link below.

This system has been running fine for years. The only change I've made is recently installing TrueNAS Core 13-U5.1 and U5.2. I do not recall this happening with U5.0. And I'm not pointing a finger at TrueNAS, its just a data point to consider. ESXi was last updated about 5 weeks ago to build 21686933 and I don't think it's ESXi.

I have examined the TrueNAS /var/log/messages file, it only shows a routine event at midnight and then the system booting up when I powered the UPS back on. I have examined some of the ESXi logs and that was Greek to me.

I also examined the UPS status using (upsc ups) to list the data, nothing odd there either.

My current plan of attack is to swap the UPS unit with my personal computer. Both are CPS 1500VA units, that is about all that is similar. This will allow me to rule out or confirm the UPS is at fault. It could be the UPS batteries for all I know however I have not lost building power during the period TrueNAS stopped recording log data so if it were a battery, external power should have kept it going. Maybe the UPS has a failed internal component.

I'm posting this in case someone has experienced this before or has an idea on a path forward. Is there any specific log data in TrueNAS or ESXi that might shed some light? For now I will swap out the UPS units and see what happens.

Thanks for humoring me.

 

[Constantin]

Does a TrueNAS system complain if it does a unintended complete, sudden shutdown w/o the usual parade of swap, etc getting destroyed?

To get the exact time for the potentially-scheduled shutdown, I'd consider hooking up a

• mechanically-driven clock to see when the system does it's shutdown. At least you'd know when it happened (clock stops).
• NC relay powering a horn from a different power source- quiet as long as UPS power is present, loud when its not (it's one way to wake up).
• iPod touch or like device recording screen continuously all night
• Raspi pinging the NAS all night until it's no longer there. Etc.

 

[joeschmuck]

No need to go through that much effort. You can see from the screen shot below that the NAS stopped recording data on 2023-07-14 at 15:46:40. You can also see that the UPS had a lot of runtime left and since I have my shutdown setup for 5 minutes on battery and then shutdown, I would expect to see a drop during that time. And of course the power on was earlier today when I found the UPS powered off. I'm trying to find out if the UPS was told to power off. All the reporting charts have the same thing, recording all normal and then all recording stops at the same time. And as I said before, nothing in the TrueNAS logs to indicate anything was going on. It seems like someone pushed the power button on the UPS, but since I was the only person in the house for the past 10+ days, unless the dogs did it, or maybe a ghost, I right now assume the UPS is faulty.

One other piece of information I just saw, guess it's saying it wasn't shutdown, so it is looking like a power loss to the NAS:

freenas.local had an unscheduled system reboot. The operating system successfully came back online at Sat Jul 15 13:49:07 2023.​

2023-07-15 13:49:07 (US/Eastern)

The little burp of lost data on 09 July was from updating to 13.0-U5.2, so that is expected.

Tomorrow I will swap the UPS units as I indicated in my first posting. I can also run a self-test on the UPS from my computer, maybe it will fail. It just has to be the UPS at fault.

I do appreciate the comments though.

 

[Constantin]

Sounds like something unhappy with the UPS then. I agree that swapping the UPS' one for the other is a good start. I wonder if the time that the system died coincided with scheduled self-tests by the UPS? What's weird is that when such tests fail the UPS usually goes into meep-meep mode and lets you know the battery is dead. Best of luck!

 

[profwalken]

Well , i get the same behavior with an Eaton PRO 1200VA , it stops alone each 7 days , not sure if UPS is making a test cycle alone and during this is going down.
once down, if i push the ups button to restart UPS then it stays up few seconds and stops once again, the only way to make it up is to switch off the plugs behind ups and restart with near no charge and once start phase is done, i can switch on plugs and it works fine.

My point is most probably the battery are near fully dead and I need to replace them by news.

Perhaps if your UPS is on with same batteries for almost 5 years , you can bet they are in same state as mine and you take no risk to replace them.

Also seen some messages pointing to defective soldering on the internal board, and just resilver it with iron can save the ups

Nota: I already got 2 others EATON doing same behavior, and each time it was dead batteries, after replacement all went like a breeze

 

[Constantin]

It could also be a dying switchmode power supply taking the UPS down with it.

Many UPS’ are priced somewhat like disposable razors, it seems. Going off-brand re replacement batteries can help, however. Mostly, those are motorcycle VRLA batteries until you get into the big stuff.

I’ve considered rolling my own but ultimately decided against it. I’d really like to see a solution where the PSU is part of the UPS to avoid the wasted power conversion.

 

[Patrick M. Hausen]

Don't your UPSs have a web interface or some management software that will let you run basic diagnostics, show the state of the battery, etc.?

I'm not buying any kind of gear anymore, that doesn't have an Ethernet interface for management.

 

[joeschmuck]

Don't your UPSs have a web interface or some management software that will let you run basic diagnostics, show the state of the battery, etc.?

I'm not buying any kind of gear anymore, that doesn't have an Ethernet interface for management.

I do not actually know how to test the UPS through TrueNAS. On my personal computer I do have that software and plan to run the tests. I just tested my personal computer UPS, passes with flying colors. Time to shutdown the ESXi server and TrueNAS along with it. I have a major electrical storm right now so it's good timing and I'll swap those UPS out tomorrow.

I hope it is a battery issue now, they don't get broken during shipment as much as an entire UPS.

 

[joeschmuck]

Well I finally swapped out the UPS units, the one for the NAS looks to have possibly one bad battery in it, although it's not conclusive. Both batteries were HOT, a symptom of constantly being charged. One battery reads 13.4V, the other 10.5V. I know the 10.5V is too low but when I read the voltage across both I read about 24V. With the UPS disconnected from Wall Power, when I turn it on, the unit comes on and goes of immediately, and no loads plugged into it. The voltage across the batteries drops slightly to ~20V before the unit powers off. I'm letting the batteries cool off and will test them separately using some other load to see if they are any good. The heat alone would have me replace both, however I have not eliminated the UPS hardware itself. If the batteries do not hold a charge, I will order new ones. If they are holding a charge then it's a new UPS. I'm almost positive it's the batteries.

Right now I suspect the one battery is causing the issue and killing the second battery in the process of overcharging. In the meantime I will be looking for both batteries and a new UPS. For now, the NAS is on a reliable UPS, my personal computer is on no UPS, just a little surge protection with the UPS batteries removed and plugged into the non-battery backup outlets.

To be honest with you, I've never seen an UPS power off so quick with a minimum load (my PC) during the UPS Self-test. It was instantaneous. And I've had a lot of UPS batteries go bad in the past and only one UPS itself fail hard.

Oh, last thing, the UPS was purchased April 2019, so 4 years of life. I paid $158 USD for it. Time to see what is on the market.

EDIT: I just placed an order for two batteries for $45 USD. If the UPS is bad then I will find out on Wednesday when these arrive. If I do have to buy a new UPS, these batteries will be available for when I need to replace the next set of batteries in a few years. A new UPS cost $200 USD from a local store. I've done mail order UPS units before and they tend to be broken when they arrive due to being heavy and delivery people dropping them, hence purchasing locally. It's worth the extra money to save in aggravation.

 

[joeschmuck]

I'm glad I spent $45 USD for the batteries, saved me a lot of money. Needless to say, it's fully operational again. It was such an odd failure, it acted more like the batteries were not there at all. Very dead.