When Disaster Comes Calling

There are times like this when I can’t help but wonder about disaster recovery plans. A large number of companies that I have worked at or spoken with over the years seemed to pay little more than lip service to this rather significant elephant in the room. This came to mind today while I was reading about the storm that ran roughshod over Toronto. In the midst of all the flooding I read about the servers at Toronto’s Pearson airport (YYZ). They had become, well, rather wet. There was “flooding in server rooms.” according to their tweet July 8th at 9:16 pm.

This really got me thinking as to how this could have happened in the first place.

At one organization that I worked for the role of disaster recovery planning fell to an individual that had neither the interest nor the wherewithal to accomplish the task. This is a real problem for many companies and organizations. The fate of their operations can, at times, reside in the hands of someone who is disinclined to properly perform the task. 

Of course this is not a truism of every company. But, there are many instances where it is the sheer force of will of the staff needed to restore service in the event of an outage. One company that I had worked for suffered an SAP outage that made it such that invoices could not be paid. The impact of this was a massive financial burden and it took the better part of a month to sort out. There was no disaster recovery plan. There was no system back up. There was no failover. In this case the DR plan was not even in existence. Through the Herculean efforts of the staff, invoices were paid manually. 

A second example that I can’t help but pull from the archives was when I was working for a certain power company. It was the end of the day and I was heading for the door with my coworker. We came upon our head of IT operations and one of the building security guards working feverishly to contain a water leak in the janitorial closet. The faucet would not close. We dropped our bags and pitched in to help.

In relatively short order the tap sheered off from the wall and the real flooding began. The difficulty that presented itself in short order was that the main water shut off valve was no where to be found. There was no disaster recovery plan that covered this contingency. To make matters worse, the computer control room was located on the floor directly below the janitor closet. 

Um, yeah.

Ultimately the situation was resolved and the control room was saved. But, it should have never gotten to that point.

So what is the actionable take away to had from this post? Take some time to review your organizations disaster recovery plans. Are backups taken? Are they tested? Are they stored offsite? Does the disaster recovery plan even exist anywhere on paper? Has that plan been tested with the staff? No plan survives first contact with the “enemy” but, it is far better to be well trained and prepared than to be caught unawares.

Even if you’re not directly involved with the plans in your shop be sure to ask the question. Are we prepared?