Energy's Response to Katrina

Storm tests the New Orleans utility's emergency plans

Three weeks after Hurricane Katrina, New Orleansbased Entergy noted that it had restored power to 80 percent of its 1.1 million affected customers. Entergy, which serves 2.7 million electricity customers in Louisiana, Mississippi, Texas and Arkansas, tests its disaster recovery plan every year, says Ray Johnson, CIO of the $10 billion utility. But Katrina delivered a worst case—a major hurricane followed by flooding in New Orleans. Entergy expects the recovery to cost up to $1.1 billion; its Entergy New Orleans unit filed for bankruptcy on Sept. 23. (On Sept. 24, Hurricane Rita caused outages in Texas and Louisiana for 765,000 customers.) Here, Johnson recounts his efforts to recover from Katrina:

Friday, Aug. 26: This evening, as the hurricane headed toward New Orleans, Entergy decided to activate its disaster recovery plan. It calls for some preliminary actions to be taken beginning 72 hours before a Category 3 hurricane is scheduled to make landfall.

Saturday, Aug. 27: At 5 a.m., Johnson sent his first IT "away" team to Entergy's disaster recovery site in Little Rock, Ark. Entergy's primary data center is in Gretna, La., across the river from its headquarters. Although the backup generators for that data center had never failed, Johnson worried that they could go down, as the storm was targeting New Orleans. The company implemented its disaster recovery plan.

Sunday, Aug. 28: As the morning began, Johnson declared an emergency with Entergy's vendor SunGard so SunGard could reserve capacity at Entergy's disaster recovery facility should it not be able to replicate systems in Little Rock.

Johnson made it to Entergy's storm command center in Jackson, Miss., around 4 a.m. Katrina had strengthened from Category 3 to Category 5. Johnson's team prepared the systems that would be most critical in restoring electricityits outage recording and management applicationsto run off the Little Rock data center in case something happened in Gretna.

Monday, Aug. 29: By 3 a.m., Gretna had lost commercial power and its backup generator was sustaining serious damage from wind and debris. The Gretna center suffered roof and water damage as well.

Tuesday, Aug. 30: Electricity was out everywhereeven in the Jackson command center. That evening Johnson sent an expeditionary force, which included some Entergy IT staff and some staff from its outsourcing vendor SAIC, to the Gretna data center.

Wednesday, Aug. 31: Although Entergy's most critical applications were brought online in Little Rock from backup tapes sent over the weekend, the team determined they could get the Gretna generator back online and bring in a generator from another facility as backup.

Thursday, Sept. 1: Entergy brought in a contractor to patch up the Gretna generator's roof, and by Friday it was running again. But with Hurricane Ophelia brewing, Entergy continued on its path with SunGard to keep its options open. Ophelia took a different tack and by Labor Day, "we had all but completed our disaster recovery plan," recalls Johnson. Critical and medium-priority applications had been restored at Gretna. Work began to restore all "normal" systems.

Preliminary postmortem: The disaster recovery plan worked well, but not without some changes. For one, Entergy's below-ground natural gas infrastructure is not typically hurt by hurricanes. But the extensive floods led to gas leaks. Systems that track gas facilities had to be made a priority.

Finding employees was essential. Entergy located its 2,800 New Orleansbased workers in the few days after Katrina using a network of supervisors. There were no deaths, though many employees' homes were damaged.

Entergy headquarters is close to the Superdome, surrounded by six feet of floodwater. Johnson and other senior executives found a temporary home in the former Worldcom headquarters in Clinton, Miss. The business continuity team helped available workers find temporary housing there.

By Sept. 9, Entergy supervisors began to build teams based on geography and skills. "Almost no one is doing the job they had before," Johnson says.

Copyright © 2005 IDG Communications, Inc.

The 10 most powerful cybersecurity companies