In August of 2003 it was just after 4 pm and I was leaving a vendor event where I was watching a professional tennis match. I was looking forward to the weekend ahead with a light Friday on the schedule. I could not have known how wrong I was and then my cell phone began to ring. My boss was on the phone. The street lights ahead of me had gone out. That wasn’t the harbinger that in retrospect it should have been.
Boss: “Get in to the office. The power has gone out."
Me: "For the office?"
Boss: “All of it"
The phone then went dead and with it the northeastern part of North America went dark. The lights out. It would be a good seven hours before any lights would come back on again.
When I got in to the office the lights were on and the systems were on and the massive diesel generators were roaring away outside. I’d seen enough tests to know why they were running. It was an ugly scene. A horrible thought came to mind. How long were these generators going to run? How much fuel did we have? It turned out that we had enough but, we had a nightmare of a time trying to get our supplier to send more diesel to our backup site. In the end they got there but, not before there was a few missteps along the way.
I can’t even imagine what that would have been like if that had all happened in the winter time. The one thing that I stood out during the entire black out was that they cell towers all began to fail. One by one they dropped. Their backup power supplies were failing. This is a problem that seems to remain today as we jump forward 13 years to present day.
Last week during a fire in New York it appears that cell towers suffered a problem like those that we saw in 2003.
From Washington Post:
Sprint generators kicked in, feeding off small fuel supplies at the site. But the generators later ran out of fuel because of a broken or stuck valve on an underground line that should have tapped diesel from a larger off-site reserve, according to local government officials briefed on the failure.
A battery-powered backup system started but also ran down, said Brian K. Hedlund, president of Sprint’s D.C., Maryland and Virginia region.
My first question would be had they been regularly testing these diesel backups? We would test ours on a monthly basis to ensure that we never had to wonder. We had learned what could have been a hard lesson in 2003 and thankfully our diesels worked. How did a failure like the one that Sprint experienced come to pass? A subject for a longer investigation no doubt.
Not meaning to jump up and down but, what are your disaster recovery plans? Have you tested yours extensively and regularly? Many times in multiple organizations I have heard the “smoking crater” scenario in the event that a meteor slammed into a data center. But, the smaller failures can be far more insidious not to mention realistic.
Does your enterprise rely on diesel back up generators? When was the last time they were started up? Might be a good time to test them and review your disaster recovery plans.