A few folks were complaining this morning that they couldn’t get on the VPN at work. I checked the phone lines, and they seemed to be out too. I figured it was a telco issue. Got Bitter up, because we were going to head to L.L. Bean because I am in great need of pants (I can’t be stealing Robb‘s gig now can I?). I figured I’d stop by work on the way there.
Once I got there, I realized we had a power issue. Nothing in my data center was on. Called up the facilities guy, and got him out there, along with the landlord. We had to get the subzero freezers out to outlets with power. All the lab plugs were dead, and the subzeros keep things at -40C for a reason. Normally they were powered off the UPS. What happened?
UPS is totally dead. Nothing on the screen. Circuit breakers all tripped. These aren’t small breakers, like you have in a residence. It’s 200 amp 480 volt three phase power. Takes a good bit of force to close them. I try to reset one, and I hear a loud arcing sound. Uh oh. Called up the UPS company. Meanwhile, we went and flipped all the breakers over to bypass the UPS and get everything on PECO. I brought up part of my systems until I could figure out what was up with the UPS. Went to L.L. Bean in the mean time.
Get back to work, and the UPS guy says it blew a transformer. It’ll take some skill and time to repair. Figuring we’ll be on PECO for a while, I brought up the rest of the room. I’m hoping all the lab equipment is OK. Those freezers were off for a while. We also have incubators for cell lines, but those are water jacketed incubators and hold their heat very well. Hopefully everything will be fine on Monday. At least all my systems were fine.
Sounds like you need some extra monitoring setup.
My monitoring is powered by UPS. Facilities monitoring should have caught the power. Not sure why it didn’t.
Isn’t that the kind of thing that triggers mass paging (Facilities monitoring, since yours was presumably down)? Which is why TAP is still on our supported systems list at work
It should have. Not sure why it didn’t. When I called was the first the facilities guy found out about it.
Heh … vaguely reminiscent of a morning at work (911) for me.
It was a Saturday morning, still early-ish, so not real busy. We were doing our usual Saturday things – reading papers, clipping coupons, drinking coffee, chatting about whatever.
*click* – and I swear there was a single loud click – Every. Single. Electrical. Device. Shut. Off. Lights, HVAC, every single one of the consoles (each consisting of three or four computers and at least six monitors) … just … gone. Phone conversations cut off, radio traffic cut off, everything. I hadn’t realized how loud this room is until that morning when it got REAL quiet.
We all kinda looked at each other … and the supervisor grumbled … and about two minutes later, the generator finished coming online and power came back.
Apparently, someone neglected to sign the maintenance contract on our room-size UPS, which is supposed to run all the critical stuff for up to an hour or something like that. It had died without notifying anyone, and the genset took its usual 30-45 seconds to fire up and another minute to get up to load-bearing speed.
I believe they’ve fixed the UPS and gotten the contract in order since then… ;-)
Get your lab freezers alarmed for temperature. Have the alarms monitored by an alarm company that has a long list of people to call until they actually get a real human who can check them out. We had a researcher lose decades of work due to a freezer failure on a long weekend. The $500 per freezer is worth it.
We’ve had dozens of incidents like this at facilities supposedly on UPS. I’m in telecom, not MIS, so not sure what the issue(s) is/are. I will say I’m not impressed overall with UPS…
I’m sure the “lessons learned” meeting will be *fascinating*
One of the larger power issues we had was janitors plugging their floor polishers into the UPS outlets.
We had them labeled in English and Spanish, they are the only outlets that are red and they still did it.
Cultural problem I think.