Iowa’s Floods: Tragic Lesson in Business Continuity

As mid-westerners recently discovered, natural disasters strike without warning, snuffing out lives, homes and businesses.

Deadly storms—including tornadoes and flooding—that ravaged the area last month hit too close to home for Deb Hale, security administrator at Iowa-based telecommunications provider Long Lines. But from the tragedy came valuable insight into the art of business continuity.

Hale’s company provides telephone service, cable TV and Internet services to a number of small communities in northwest Iowa and wireless service to communities in South Dakota, Iowa and Nebraska. The customer base consists of business as well as residential, and the 300-employee company provides 911 services to some communities as well.

Given the critical infrastructure involved, many organizations would suffer dearly if a disaster forced Long Lines to cease operations.

Hale, also a volunteer for the SANS Institute’s Bethesda, Md.-based Internet Storm Center (SANS ISC), recently shared her experiences in a write-up on the ISC website. In this Q&A, she discusses the most important things a company can do to survive what Mother Nature decides to unleash.

CSO: Your company is located on the other side of the state from where the flooding happened, but was there any collateral damage in the form of service disruptions and the like?

Deb Hale: We have been pretty fortunate. We had two vendors located in the Cedar Rapids area that we receive a lot of support from. One of the vendors had multiple locations within the state so they were able to transfer phones, services, and so on over to the other locations in the state that were not impacted. Another vendor provides a service to us via the Internet. This company had the good fortune to have a president and company founder that understood disaster planning and so they had redundant systems. With one gone, the other jumped in and took the load.

I’m sure communication has been a problem, though.

The biggest issue we had is that it took a little longer to get a hold of them. This was due to the fact that many of the employees for these two vendors had personal losses of home and property and were attempting to deal with these losses and the cleanup involved.

In your SANS ISC diary entry, you mentioned one vendor in particular who was hard hit. Who was the vendor and what might their situation mean in terms of the quality of service you get going forward?

The vendor referenced in the article is a service provider. They supply a product which we then resell to our customers. The product is a software system that protects our customers’ computers from virus, spyware, adware, and other exploits/compromises. They are currently up and running again at another location in the Cedar Rapids area. We have had no problem or complaints from our customers about their level of service and we have not noticed any problems either. I have their application suite installed on four of my personal computers and I am pleased to say they have not lost connection to the “mothership” at all.

You mentioned that the tornado that hit the Boy Scout camp was closer to where you are. Does your company have any kind of preparedness plan for how to protect employees and infrastructure in the event of a tornado or other event?

The location of our company has many potential hazards, being close to a highway, major interstate system, airport, military facility, railroad tracks, rivers, and so on. We are very fortunate to have leadership that understands what disasters are all about. Many of our employees are volunteer EMTs, firefighters, response personnel, and one of the owners is a member of our USAR (Urban Search and Rescue) Team. We attempt to be on top of things and have much of our equipment spread out in differentlocations throughout our service area. We are currently working on a comprehensive plan which will include a bunker facility. The bunker facility will house redundant equipment and resources. With that we will be providing offsite backup service to businesses within our customer base.

That said, are there any fresh lessons you took from the disaster in terms of your own business continuity plan and any weaknesses recent events may have revealed?

I am a member of the Safeguard Iowa Partnership. We had a review meeting recently to discuss some of the things that went on during the response phase. A couple of items that have already been discussed is that some partners even though they were on high ground and had no flooding at their facility were unable to get to the facility due to the flood water over all of the access roads. And many of them had residual impact due to their vendors or customers being directly impacted. We will be meeting in thenear future to discuss the lessons learned in depth. I am sure that there are going to be many.

To ask the same question you ask in your diary headline, would your business survive?

We have already put a lot of pieces in place to allow us to operate out of one of our other facilities. We still have a ways to go but we are making headway. Would we survive? I hope so, but until our plan is complete, until we have all of the redundant systems in place, until our testing is complete I do not know for sure. Unfortunately there is no way to plan for every possible disaster and there is always the chance that something may go wrong. I certainly hope not but realistically I can make no guarantees. All we can do is develop the plan, test the plan, update the plan, test theplan, update the plan – you get my drift.

One of the big lessons that I have learned over the years is that a disaster recovery plan/ businesscontinuity plan is a living, breathing entity. What we have today is not what we had five years ago, even two years ago. We have to continually be thinking change and how it impacts the company and its ability to continue to operate.