Disaster Recovery: Write People into the Plot

Call center operator MM Teleperformance has plenty of practice dealing with disaster and disaster recovery. In recent times the Bristol, U.K.-based company—part of a French-owned group that operates 179 call centers in 36 countries—dealt with the aftermaths of both September 11, 2001, and Hurricane Ivan, and advised U.K. citizens on travel to the Middle East in the run up to the Iraq invasion of 2003. Retained by the British government to field emergency calls in the event of a disaster, the company is required to have fully briefed agents on standby to field calls from the public within four hours of a service invocation.

But the Dec. 26, 2004, tsunami brought disaster closer to home. Although a call center that the company operates in the Gurgaon suburb of Delhi, India, was physically unaffected by the catastrophe, the sheer scale of the devastation forced management to ponder a number of unpalatable what-if scenarios. The center employs a considerable number of people from southern India, for example, and many of them

had taken advantage of the holiday to visit relatives—some residing in the affected regions. Ultimately no MM Teleperformance workers were killed, injured or delayed in their return, but center managers wouldnt know this until much later.

“We recognized that we needed to look again at our disaster recovery plans,” says Chief Operating Officer Bibi Bajwa. “It was important to know that we had a workable contingency plan in the event of a full-scale disaster.” Long-held nagging doubts about the nearby backup center, for example, have now crystallized—especially given the proximity of both buildings to the Delhi airport. “I dont now think it is sensible to have the two so close together,” she says. “When we can, well change it.” The disaster recovery plans for the companys two operations in the U.K.s Midlands region, for example, call for core operatives to be quickly bused to an unaffected site, from which a skeleton service would be operated—responding to incoming calls only, for example, rather than making outgoing calls as well.

Bajwa is not alone in contemplating the consequences of disaster with a capital D. Time and again, notes John Medaska, a vice president with IT services vendor Relational Technology Services of Tampa, Fla., supposedly workable contingency plans have foundered in the face of devastation that is widespread rather than localized. The weak spot? The human element.

“With their homes and neighborhoods in ruins or under threat, employees—as youd expect—tend to concentrate on their personal life. But because thats never been considered at the planning stage, companies are surprised when it happens,” says Medaska. In the immediate aftermath of 1992s Hurricane Andrew, he adds, employers surveys apparently revealed that up to 70 percent of employees who were working on supposedly mission-critical tasks didnt show up for work. “Their planners put their hands up and said, Gee, we never imagined that people would put their wife and kids before the company,” notes Medaska, wryly. And so, minutely detailed business continuity plans might contain a gaping hole, if people arent on hand to carry out those planned actions.

Read on for tips from companies with lessons to impart on ensuring employees are as well taken care of as IT systems or other business assets.

Employees and Disaster Recovery: Rising to the Occasion

First, a caveat. No two disasters are the same, and employees responses will differ. While blanket no-shows certainly happen, experience suggests that this is exceptional. “In the face of adversity, human beings will usually step up to the plate,” asserts John Wisbey, head of business continuity services for Europe, Middle East and Africa at Computer Sciences Corp. And Wisbey knows of what he speaks: He was personally involved in the aftermath of two high-profile terrorist atrocities in the United Kingdom—the Manchester city center bombing of June 1996, and the bombing of Londons Moorgate in April 1993. The Moorgate bomb, which struck right at the heart of Londons financial district and caused almost $1 billion of damage, has especially colored Wisbeys views. An employee of financial institution J.P. Morgan at the time, he witnessed firsthand how people and organizations pitched in to help shattered businesses get back on their feet. Thanks to borrowed equipment and freely given help to install it, the trading floor of the Saudi International Bank, for example, was back in action just 48 hours after the bomb detonated, recalls Wisbey, noting that similar reports emerged from New York City in the aftermath of 9/11.

“People are far more resilient than is often imagined,” he says. Yet while recognizing that resiliency, Computer Sciences Corp.s own business continuity plans make practical allowance for the unpredictability of disaster. Given an event that necessitates employee relocation, he explains, the company predicts that the majority of its employees would rise to the occasion, even in circumstances of widespread disruption. Despite this, there is an acceptance that some employees would have other priorities. “A certain percentage would put their families first, sure,” says Wisbey, but for genuine reasons, which would be accepted as such. “It could be you next time; people recognize this,” he says.

So while employee resilience is a useful bonus to compensate for vagaries in response, its better regarded as an adjunct to the plan, rather than the plan itself. A similar seasoned pragmatism underpins the business continuity philosophy of consultant Michael Arata, who until this spring served as director of corporate security and facilities for a large construction company based on the West Coast. “Here in California, earthquakes and natural disasters are a fact of life,” he says. “To employees, the family comes first; people just arent going to show up. You can order them to, but they still arent going to show.” And even if employees wanted to head for the office, they might be prevented from reaching it. “When an earthquake strikes, the authorities dont want people on the roads,” says Arata. “Bridges and underpasses are closed until they can be inspected for damage, and roads are kept clear for emergency vehicles. You just cant assume that people will be able to make the journey.” Likewise, he adds, you cant assume that any employees at work when an earthquake strikes will be able to get home. “We have a provision within our disaster plans to keep stranded employees safe and fed, even in the absence of power supplies—you cant just throw people out onto the street,” he says. Consequently, business continuity plans at Aratas former employer have a two-pronged approach. A small corps of people at corporate headquarters is nominated to keep critical services like IT working—if possible. But fairly implicit in the assumptions that underpin the plan, adds Arata, is that this will in practice prove problematic. Accordingly, the construction companys offices in other California locations are primed to take over if required. Important though this flexible approach is, this aspect of business continuity planning isnt what most engages Arata. A personal experience cemented Aratas more closely held view. Visiting another company facility in midtown Manhattan on the morning of September 11, 2001—one with an excellent view of the twin towers—it was to be eight weeks before he returned home. Employees had gathered to watch the smoke and flames billowing from the first attack, he explains. “Then the second plane hit, and everyone went to pieces. We didnt have anybody killed, but we certainly had people affected through knowing other people who had been killed,” he says. Basic needs were simply met. As New York isnt an earthquake zone, there were no stocks of provisions, says Arata. The solution: The cafeteria stayed open, feeding people until they were able to make their journey home through the chaos. Longer-term needs, though, have been met by counseling. “Its one thing thats been learned from September 11th: Your plan must include a provision for counseling,” Arata says. “People really are the business, and you cant ignore them.” Keeping in Touch Few businesses factor their people into business continuity planning as thoroughly as Morristown, N.J.-based telecommunications giant AT&T. When disaster strikes, says Director of Business Continuity and Recovery Services Jerry Shammas, the company aims to be ready to respond—and to that end it not only holds full-scale practices four times a year but also subjects those practice events to deliberate extremes of climate. “We need to know that our plans will work anywhere in the U.S., under any weather conditions—irrespective of whether theres three feet of snow in Pittsburgh, or its a hot humid day in Los Angeles,” he says. “We also invite state and local disaster recovery agencies to participate, as well as local telcos. They like the opportunity to practice, and it helps make the scenarios as realistic as possible.” Although the details of AT&Ts plans differ slightly depending on whether it is a data center or a communications facility that has been affected by a disaster, explains Shammas, the basic procedures are the same. At four locations spread across the continental United States, 120 trailer-loads of network recovery equipment lie ready to be sent to a disaster-affected site—diesel generators, network equipment, tools and replacement communications equipment. (And yes, those four full-scale practice events each year do involve actually dispatching a subset of the trucks to the site of the mock disaster.) “The idea is that the trailers can be onsite within 24 hours, subject to federal and state authorities allowing access,” Shammas says. Upon arrival, he explains, the trailers are met by a team of network recovery engineers who have been flown in to get the equipment up and running. Critically, the starting premise is that there wont be anyone else around to help. “We dont rely on local employees at all,” says Shammas. “We assume they arent there.” But it isnt a case of “out of sight, out of mind.” Even as the trucks are getting under way, an incident recovery team starts efforts to make contact with every employee who works at the affected site, using their home phone, cell phone or other known means of contact. And if they cant make contact that way, an employee from the network recovery team—when it arrives onsite—will be dispatched to their home address. “We try very hard to get in contact with everybody,” says Shammas. “We want to make sure that they are all right, and identify what kinds of support they might need in terms of food, clothing and shelter. If they need cash, well lend them cash. If they need accommodation, well put them in a hotel or motel, or put them in contact with the Red Cross. We also have trailers that can be fitted out as conference rooms and offices, but which can also, if required, be fitted out with beds. The basic idea is to help people get back on their feet.” Which is always easier if people havent been knocked down in the first place, of course. Shammas says the company tries to anticipate likely disasters and prepare accordingly. If a hurricane were to move along the Florida coast, for example, AT&T facilities in its likely path would transfer as much as possible of their workload to other locations, and send their workforces home. “If we know that something like that is going to happen, we have our people go home and look after their families, or board their windows, or go somewhere else,” says Shammas. Many of these recovery planning and coordination functions were centralized under a three-person Incident Management Operations Center group, reporting to the COO; recently AT&Tdisbanded that center and absorbed the functions into other groups. What of relocating employees to backup facilities? The answer depends on the scale of any disruption, says Shammas. If a disaster is highly localized—a fire or flood, for example—then employees will be expected to relocate temporarily to the backup site. AT&T will provide transport if required, or employees can use their own. But if the disruption is wider, then other employees are repositioned to take up the load. “It depends on the scenario involved and if the local environment and community are affected,” says Shammas. Going to Extremes AT&Ts approach to human factors ought to pass muster with John Medaska. His gripe, youll recall, was that too many companies never even consider the human factor in their contingency plans and are consequently caught out when disaster strikes. But Medaskas own definition of a gold standard is a tough one to beat. An unnamed financial institution, it turns out, has a data center in Florida. Concerned about the difficulties of relocating employees to a backup facility in a region known for hurricanes, the company analyzed historical meteorological data for risk factors. The outcome: The probability that two locations a hour and a halfs drive time apart would be affected by the same storm was extremely remote. And an hour and a half, it figured, was not only a distance that seemed about right in terms of managing, keeping current and testing a backup facility, it also was about the farthest that people would want to commute. Not that commuting is the preferred option. Near the location of the backup facility, the company also has in place contracts with local hoteliers. Not only can rooms for key staff be made available at very, very short notice at prearranged rates but they are also suitable for long stays, families and pets.As disaster planning goes, says Medaska, this companys plan is pretty hard to fault. Trouble is, as best practices go, its also pretty rare.