• United States



by Senior Editor

Business Continuity and Disaster Recovery in a Tornado Zone

Sep 03, 20098 mins
Business ContinuityDisaster Recovery

Tornadoes create risks that differ in key ways from those in a hurricane area. Cancer Treatment Centers of America showcases a study BC/DR plan for dealing with tornadoes

Business continuity planners in tornado alley have much in common with those in hurricane areas — but also key differences. Tornadoes have smaller funnels, but can appear in groups, may feature dramatically higher winds, and can strike with far less warning than a major hurricane typically provides. Good BC/DR planning must take those differences into account in everything from employee and facility safety to network uptime.

Also see: BC/DR Challenges in a Hurricane Zone

Cancer Treatment Centers of America literally has lives on the line if something goes wrong with their business continuity plans. That’s why Chad Eckes, chief information officer of the Schaumburg, Illinois-headquartered organization, believes there is no room for complacency. Keeping things running smoothly 100 percent of the time is always the goal.

“I think you will find most CIOs will say 100 percent up time is impossible,” he said “But why would you ever target anything less than 100 percent? We have always operated well over five-nines in terms of up time. But if I were satisfied, those potential moments of down time could be the moments a patient is being impacted.”

The mission of Cancer Treatment Centers of America, according to Eckes, is to offer healing and hope to complex cancer patients. It’s a goal that everyone with the organization is mindful of, regardless of their job, he said.

“Everyone at CTCA draws a line everyday in how their job touches the patient.”

CTCA has hospitals in Arizona, Illinois, Oklahoma and Pennsylvania. But CTCA’s primary operations in Schaumburg are in tornado country, which means keeping an eye on the weather and having a business continuity plan that is resilient in the event of a damaging storm, particularly in the spring, when tornadoes are most common in this part of the U.S. The Midwest and the Central Plains of North America see more tornadoes each year than any other part of the world.

“We had a key decision to make a few years ago about whether we run centralized data centers or decentralized. Going back to that core principal of the mother standard, we made the decision by bringing it back to patients and its impact on patients.”

Eckes and CTCA decided centralization was the best option so the same information was available to all employees, regardless of the facility they are working in. From a BC/DR standpoint, Eckes said a centralized data center with a backup facility was the better way to go in order to avoid any down time in the event of an emergency that takes one of the data centers offline for any period of time.

“We have migrated to all digital. There is no paper backup. We have our bedside monitors directly connected into our electronic health records. Our phone is all VOIP. Paging is integrated into the phone system. If any of these core systems go down, it could be a patient’s life. You can’t call a code blue if your phone system is down. It’s that critical that everyone takes this that seriously.”

According to a survey conducted earlier this year by AT&T, the most common scenario for rolling out a BC/DR plan was extreme weather (Read more about the results in Survey: BC/DR Plans Factor in Mobile, Social Networking). About 25 percent of companies said weather forced their plan to be put in action. More than half of companies in Houston and Miami/Orlando/Tampa, regions that have been hit hard with hurricanes in recent years, were likely to have invoked their business continuity plan.

But unlike hurricanes, which are a type of storm system that often originate over tropical waters and come with much advance warning, tornadoes are isolated storms that form with less warning for weather officials. They form in moist, warm air in advance of a cold front and are often seen in their hallmark funnel-cloud shape; a violent, rotating column of air which is in contact with both the surface of the earth and a cloud. Tornadoes can stretch more than a mile across and stay on a destructive ground path for many miles, wiping out structures and picking up objects and debris along the way.

Also see 5 Ways to Build a Business Case for Business Continuity

With tornado patterns in mind, CTCA built their two data centers in greater Chicagoland so that they sit 59 miles apart and in a pattern in which the likelihood of tornado hitting both of them is nearly impossible, said Eckes. The locations were chosen based on information CTCA got from the Federal Emergency Management Agency (FEMA) about weather patterns. The decision was based on historical events and what the likelihood would be of natural disaster hitting both facilities. Eckes said CTCA made sure the facilities, which have identical data, were sitting in a north-south arrangement and more than 30 miles apart to ensure one facility would always be operating. According to the National Oceanic and Atmospheric Administration (NOAA), tornadoes typically move from southwest to northeast.

“The first main design from a BCP standpoint was to have complete redundancy in our data. Anytime there is any production data written to the primary it is immediately mirrored over to our DR data center,” said Eckes. “Literally, we are up to date in our second center within 15 seconds. That is, with a complete copy of all clinical systems.”

The CTCA data has five layers of redundancy, from mirroring of data with an EMC SAN, to disk backups, to snap shots of patient data that are taken every four hours and stored on local servers within each hospital.

“If our redundancy fails in terms of our network back to our two data centers, we need to have that data in the hands of our clinicians,” said Eckes. “So they can go to this centralized server in their hospital, print out the pdf or copy it to a PC and still have all the information they need, like allergies, medications, treatment clinical pathway. Everything is there for them to care for their patients.”

Structurally, considerations were also made due to the possibility of tornadoes hitting the data centers. While hurricanes can produce high winds that cause damage, preparations in hurricane zones often consider flooding as the most damaging potential factor. However, a tornado does most of its damage with extremely high winds; tornadoes can generate violent wind speeds in excess of 250 miles per hour. (For comparison, a category 5 hurricane may hit land with sustained winds of 155 miles per hour; Hurricane Camille at 190 mph was one of the most intense in history.)

Tornados are measured by the Fujita scale; a system which assigns levels of destructive power based on post-storm assessments. This scale runs from an F-0 storm, which causes little to no measurable damage, all the way to an F-5 which can completely eliminate all structures in its path. While most tornadoes rank lower than F-3, according to FEMA, they can still cause damage to a facility, particularly on windows and roofs.

“One of the important things for us was ensuring there were no exterior windows facing into the data center,” said Eckes.

But Eckes said structural modifications weren’t enough to give him peace of mind, so CTCA built their recovery center in an old bank vault with the specific risk of a tornado in mind. The vault is in a brick building with a data center that is surrounded by a perimeter of 18 inches of poured concrete that is reinforced with rebar.

“The likelihood of a tornado being able to hit, even at F4 level, is near impossible,” said Eckes. “This is about as much protection as we are going to get without having an underground bunker.”

According to FEMA, the best defense for personnel that may be in the immediate destructive path of a tornado is protection in a basement, cellar or other underground storm shelter. If there is none, staff should be advised to go to the lowest floor and into a small center room such as a bathroom or closet, under a stairwell, or in an interior hallway with no windows.

Geoff Craighead, vice president of High-Rise and Real Estate Services at Securitas Security Services USA and author of “High-Rise Security and Fire Life Safety,” advises clients he works with in tornado zones to consider all physical elements of a building when created a business continuity plan.

“Creating a business continuity and disaster recovery plan requires a comprehensive evaluation of all physical factors that during a crisis may impact key business processes,” he said.

Also see Disaster Recovery: Write People Into the Plot

Tornado warnings, when they are possible, are often broadcast on both radio and television, which of course can be monitored in the average security or network operations center. Craighead said if an organization is warned there is possibility of a tornado in the near future, preparations could include securing or moving outdoor objects such as trash containers, planters, signs, furniture, and vehicles that may blow away or cause damage to people or property. Craighead also recommends pruning tree branches that may cause damage to the building if time permits. Occupants should clear all objects from desks and working areas and all exposed paperwork should be stored in closed cabinets and other containers, he said. Valuable equipment and documents should be moved from outer offices to interior rooms.

“Building management, engineering or security staff, or floor wardens may conduct walkthroughs of the building to ensure that appropriate precautions are being undertaken,” said Craighead.