The Humble UPS

[ewhac]

"EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE....!" (abridged)

That's what greeted me when I got home last night. The UPS had decided that the battery had reached the end of its life and needed to be replaced. And the way it lets you know that is by continuously going, "EEEEEEEEEEEEEEEEEEEEEEEEEEEE...!" And the only way to stop it from doing that is to turn the UPS off.

So my first thought was to get a replacement battery, but while I was searching for a place to buy one and discovering that no local retailers on the SF peninsula stock them (I'm glaring at you, Fry's), I began to wonder whether it might be a better idea to upgrade the whole UPS. My current unit is an APC Back-UPS ES 750 (model BE750BB). The unit reports a load of about 29%, and it is sized to let the NAS and gateway live just long enough such that they can shut down cleanly in the event of a power outage lasting more than a minute. I have no complaints about the unit. pfSense and FreeNAS can both talk to it and, until it tried to get my ears pierced, it had done its job admirably and quietly. Replacing the battery is the less expensive option ($40 vs. ~$100).

So: Replace the battery, or full upgrade?

 

[gpsguy]

Did you try Batteries Plus?

 

[Z300M]

"EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE....!" (abridged)

That's what greeted me when I got home last night. The UPS had decided that the battery had reached the end of its life and needed to be replaced. And the way it lets you know that is by continuously going, "EEEEEEEEEEEEEEEEEEEEEEEEEEEE...!" And the only way to stop it from doing that is to turn the UPS off.

So my first thought was to get a replacement battery, but while I was searching for a place to buy one and discovering that no local retailers on the SF peninsula stock them (I'm glaring at you, Fry's), I began to wonder whether it might be a better idea to upgrade the whole UPS. My current unit is an APC Back-UPS ES 750 (model BE750BB). The unit reports a load of about 29%, and it is sized to let the NAS and gateway live just long enough such that they can shut down cleanly in the event of a power outage lasting more than a minute. I have no complaints about the unit. pfSense and FreeNAS can both talk to it and, until it tried to get my ears pierced, it had done its job admirably and quietly. Replacing the battery is the less expensive option ($40 vs. ~$100).

So: Replace the battery, or full upgrade?

For a long time I just bought replacement UPSes when they were on sale. More recently on a few occasions I bought some replacement batteries from Chrome Battery, but they did not last all that long, and -- after the warranty had expired -- I found that the manufacturer's own markings showed a lower capacity than that on the "Chrome Battery" label that was stuck over them. My last purchase of replacement UPS batteries was of CSB batteries, the same as those over which APC sticks their own labels.

Maybe the age of the UPS should be taken into account.

If you have a Costco membership, look at the CyberPower UPSes that they sell for $100: significantly higher capacity than an ES750.

 

[ewhac]

Did you try Batteries Plus?

If you're referring to these guys, their Web site suggests they don't carry a compatible replacement for the RBC17 battery.

 

[ewhac]

...I found that the manufacturer's own markings showed a lower capacity than that on the "Chrome Battery" label that was stuck over them. [ ... ]

Yeah, I realized almost immediately once I started chasing links on Amazon that this market is probably just as flooded with off-brand junk and counterfeits of off-brand junk as is the market for Li-ion batteries for cameras and laptops.

If you have a Costco membership, look at the CyberPower UPSes that they sell for $100: significantly higher capacity than an ES750.

(*forehead-smack*) This honestly did not occur to me. And they have the APC 900VA for $20 less than everywhere else. Hrmmmm...

 

[danb35]

Most UPSs, in my experience, use pretty standard 12-volt sealed lead-acid batteries, either the 4.5 amp-hour or 7 amp-hour sizes--the RBC17 looks like just a single 7 amp-hour battery. I'd be amazed if Batteries Plus didn't have an equivalent battery on the shelf. In fact, they do; here's the link from their web site--though their "compatible" batteries are just as expensive as an "authentic" APC battery from Amazon.

Edit: Though a bigger UPS is never a bad thing--it's probably obvious that I don't subscribe to "just long enough to let everything shut down cleanly" school of thought.

 

[ewhac]

...I'd be amazed if Batteries Plus didn't have an equivalent battery on the shelf. In fact, they do; here's the link from their web site [ ... ]

Initially, I wasn't convinced that the BE750G and my older BE750BB were even physically compatible. But according to APC's own battery replacement selector, they recommend the RBC17 for both models, so that expands my search a bit.

Edit: Though a bigger UPS is never a bad thing--it's probably obvious that I don't subscribe to "just long enough to let everything shut down cleanly" school of thought.

I'm not running a 24/7 data center; I'm okay with my NAS going offline while I'm away at work. I'm just mindful that ZFS can keel over if you yank power away from it, so I wanted to ensure that didn't happen.

 

[Robert Trevellyan]

the only way to stop it from doing that is to turn the UPS off

There's probably a way to silence the alarm. At least, it seems like a standard feature of current generation models. That way you'd still have a working surge protector until the new battery/UPS arrives.

 

[Arwen]

...
I'm just mindful that ZFS can keel over if you yank power away from it, so I wanted to ensure that didn't happen.

ZFS was designed from the start to allow graceless shutdowns. That's the whole purpose of COW, (Copy On Write).

But, some people have had less than ideal experiences with graceless shutdowns. My personal thought is that it was
one of the following issues;

• RAID controller card has write caching and was lying to ZFS about write buffer flushes.
• BSD disk encryption was used, which adds a layer that ZFS does not know about.
• Misc. hardware faults, (like disks lying about write barriers or write cache flushes).
• Last, actual bugs in the disk I/O stream, including ZFS bugs :-(.

In someways, I wish we could get an analysis of systems where people lost data on ZFS. Then we could list the
most common reasons, as well as potentially fix bugs that are unknown, or known but not thought important.

 

[Ericloewe]

RAID controller card has write caching and was lying to ZFS about write buffer flushes.

Misc. hardware faults, (like disks lying about write barriers or write cache flushes).

My money is on these two. Unfortunately, this is one of those things that's a pain in the ass to debug...

 

[Adrian]

If you have a Costco membership, look at the CyberPower UPSes that they sell for $100: significantly higher capacity than an ES750.

Elsewhere I have a CyberPower CP1500EPFCLCD UPS with comms gear and a Windows PC attached. It seems to work well. As its supplied (Windows) monitor software is adequate I have not tried to monitor it with anything else.

 

[ewhac]

ZFS was designed from the start to allow graceless shutdowns. That's the whole purpose of COW, (Copy On Write).

That's what I would have thought, too. Since the advent of journaled filesystems, it seemed odd to me that something as sophisticated as ZFS wouldn't be resilient in the face of yanking the power cable. But between the ZFS-has-no-recovery-tools rule, and anecdotes here of pools trashed after a power outage, I decided, "Nah, not taking the risk." Besides, I was already in the habit of keeping my gateway on a UPS, so adding the NAS was an easy move.

I'm also concerned with power fluctuations. It's only happened a handful of times, but we've had brownouts here, and computers tend to do weird data-corrupt-y things when the power drops below spec. A UPS defends against this as well.

If you have a Costco membership, look at the CyberPower UPSes that they sell for $100: significantly higher capacity than an ES750.

After taking a look, one odd thing I noticed is that none of the CyberPower units at Costco appear in nut's list of supported hardware. I'm guessing all current-gen UPSs show up as generic USB HID devices, but it would be nice to know for sure.

 

[danb35]

FWIW, I've had a CyberPower UPS on my FreeNAS box without a problem. It, or a very similar model, was in the extensive drop-down.

Edit: Well, mostly without a problem. The USB connection seems a little flaky; there are a number of similar reports around here. It doesn't do any harm that I can tell other than generate spurious warning messages, though.

 

[rs225]

I think I'm going to start doing some cord yanking tests.

edit: just did 5 power cuts under write load on a 3 disk ashift=12 raidz1 test pool, absolutely nothing of interest happened. Would be curious to see a 4 disk test in case there is a problem with less perfect configs.

 

[StephenFry]

Would be curious to see a 4 disk test in case there is a problem with less perfect configs.

Unless something has drastically changed for the worse, I can confirm that from anything from 1 to 6 drives (tested striped, mirrored, Z1 and from 4-drives on Z2) nothing untoward will happen. Yank away.

Before I committed to ZFS five+ years ago, I spent about a year messing around with it and doing all kinds of QC checks. It's shockingly solid.

 

[rs225]

It's shockingly solid.

I think so too. The power failure timing may just be when a corruption was discovered. Typical rates of hardware glitchiness could explain that, but whenever an explanation is too convenient, I pay attention.

 

[ewhac]

Update:

Thanks to @gpsguy's suggestion and persistence, I ended up buying a replacement battery at Batteries Plus (Duracell 9VA), who had a storefront nearby work. I get it home and, after digging the UPS out of the dust bunny fort under my desk, discover that it's about 30% smaller physically than the battery already in there. It fits; it's just smaller. (*sigh*) But it's still 12V, and it's still lead-acid chemistry so, at worst, I've lost some back-up time/capacity, and maybe some maximum load (drawing the max rated load off a physically smaller battery could I suspect cause thermal issues).

After plugging everything back in, the UPS stats in pfSense weren't making any sense. "Oh," I thought, "I have to recalibrate for the new battery, don't I?" To recalibrate an APC UPS of this vintage (I found some of the original paperwork -- it's 11 years old), you have to let the battery charge to 100%, plug in a load of at least 30% of its rated maximum, then yank the power cord and let it drain out completely. In this case, the BE750BB is rated for 450W, so a 150W load should be about right. A consistent, non-reactive load is best, so the obvious choice would be some light bulbs. Trouble is, we converted over to CFL and LED light bulbs years ago. So I had to go out and actually buy a 150W incandescent bulb. But plug it in I did, and the UPS was able to keep it on for nearly 25 minutes before giving up the ghost. Nothing to write home about, but certainly enough to keep the machines from fouling their britches.