Power Up: A New Orleans Energy Company's Post-Katrina Struggle

The CIO of a New Orleans-based energy company describes the struggle of a lifetime.

“Monday was not a good day.”

That’s how Ray Johnson, not one for hyperbole, remembers Aug. 29, 2005 the day Hurricane Katrina roared ashore on the Gulf Coast. Johnson, CIO of New Orleans-based Entergy, would see darker days. But he and his team would also see success, as they struggled against the odds to restore electric power to more than a million customers.

Entergy’s recovery efforts can be traced back to long before Katrina hit. The $10 billion energy company has a disaster recovery plan that has been tested and revised once a year. Last year, that “test” came in the form of Hurricane Ivan. Entergy activated its storm command center and disaster recovery processes in response to the approaching storm, which ultimately made landfall further east in Pensacola, Fla. And this April, the company conducted an extensive storm drill which simulated what corporate executives thought of as a worse case scenario – a major hurricane followed by extreme flooding.

“This is not a unique event for us,” explains Johnson, “I wish it were. But we’ve got our disaster plan nailed. We review it at least once a year and either we conduct a drill or get to test it when a hurricane threatens us and then misses.”

On Friday, August 26, it looked like Johnson would have another opportunity to test the disaster recovery plan. Katrina was crossing over Florida and beginning to churn in the Gulf. But early in the day, the hurricane was predicted to hit several hundred miles east of New Orleans.

As Friday evening wore on, the situation changed. “We started making our storm calls and made the decision to activate our disaster recovery plan,  which calls for some preliminary actions to be taken beginning 72 hours before a category 3 hurricane is scheduled to make landfall,” recalls Johnson.

Saturday morning at 5 a.m., Johnson sent his first “away” team to the disaster recovery site that Entergy maintains in Little Rock, Ark. Entergy’s primary data center is located in Gretna, La., across the river from Entergy’s corporate headquarters. While the back-up generators supporting that data center had never failed, Johnson worried that they could go down. “By late in the day (on Saturday) it became more and more obvious as Katrina tracked closer and closer that New Orleans was going to be a primary target.”

Around that time, the decision was made to do a full implementation of the disaster recovery plan and Entergy’s core restoration team flew into action, led by Randy Helmick. During normal operations, Helmick was known as the vice president of customer service and support, but when an emergency was declared, he took on the title of “storm boss.”

Johnson made it to Entergy’s storm command center, a.k.a. “The Power House” in Jackson, Miss., around 4 a.m. on Sunday morning. Katrina had strengthened from category three to category five. “The news reports were alarming,” Johnson says. “The potential implications for the city went up dramatically.”

As outlined in the disaster recovery plan, Johnson’s team prepared the systems that would be most critical in the restoration of electricity – its outage recording and management applications to run off the Little Rock data center in case something happened in Gretna.

Sure enough, by 3 a.m. Monday, Gretna not only lost commercial power, but the back-up generator was sustaining serious damage from wind and debris. “We’ve had pretty bad storms before where we’ve lost commercial power and failed over to the generator,” explains Johnson. “But we’ve never lost both.” A day later he would learn that the Gretna center suffered roof and water damage as well. As the morning began, Johnson declared an emergency and warned its vendor SunGard to reserve capacity at its hot site facility should they not be able to replicate systems in Little Rock.

Tuesday, the electricity was out everywhere even in Jackson. “Tuesday was a pretty rough day,” Johnson says. “We didn’t even have power at the power house.” That evening was the first chance for Johnson to send an expeditionary force to the Gretna data center.

A significant portion of Jackson’s IT staff works for SAIC, as part of a major outsourcing relationship dating back to 1999. “When we sent people out to the data centers in those first few days, when there was no food or water yet, you couldn’t tell who wore what badge. It didn’t matter, we were all working together,” Johnson says.

Many vendors went beyond the call of duty. “All of our vendors – and even some we’ve never worked with – were there within a couple of days saying, What do you need? How can we help?” recalls Johnson.

Although Entergy’s most critical applications were brought online in Little Rock from backup tapes sent over the weekend, on Wednesday the team determined they could get the Gretna generator back online, bringing in a generator from another facility as backup. On Thursday, they brought in a contractor to patch up the roof, and by Friday it was up and running again. The bad news was another storm was brewing – what would become Hurricane Ophelia – so Entergy continued on its path with SunGard in order to keep all our options open. Ophelia took a different tack, and by Labor Day, “we had all but completed our disaster recovery plan,” recalls Johnson. All critical and medium-priority applications had been restored at the Gretna data center and the disaster recover team worked to continue to get all “normal” systems up and stabilized.

By all accounts, the disaster recovery plan worked well, but that was not without some changes made along the way. “We never follow the plan to the letter,” explains Johnson. “In the IT space, the plan is very solid in terms of what we have to do. But we’re always working – our core IT staff in conjunction with representatives from the business areas to see if we need to change priorities.”

Entergy’s natural gas infrastructure below ground is not typically impacted by hurricanes, as is the electrical infrastructure. But because of the extensive flooding, gas leaks followed. “That changed the game a bit,” says Johnson. “The applications associated with natural gas facilities mapping and asset tracking systems had to be moved up the list in terms of priority. It became clear that that would be much bigger part of the restoration effort.” Entergy New Orleans Gas Operations is currently working to find, control and repair those gas leaks where the water has receded.

Johnson is the first to tell you he’s one of the lucky ones – the damage to his house on the west bank of Lake Ponchatrain in Jefferson Parish can be fixed. But not everyone was so fortunate. The first few days after Katrina, Entergy was also focused on locating its 14,000 employees, including the 700 men and women working in IT. They were scattered to the four winds. “One of the first things we did was to try to contract everyone using our network of supervisors beginning on Tuesday to find out, ‘Are you ok?’ ‘Do you know the status of your home? Do you have any urgent needs that aren’t being met?” says Johnson. “There were quite a few people we couldn’t find at first.” Ultimately, everyone was located and there were no fatalities.

 “What’s unique about this story is that so many people involved in the core restoration that relocated here to Jackson were on the job working hard, even though they knew they had no home to return to,” Johnson says. “People up to the senior management level couldn’t get touch with family members, and these people were working with us twenty hours a day.”

Beyond disaster recovery, some Entergy had to work on business continuity plan as well. That commenced the Sunday following the storm. Entergy headquarters is located in the heart of the Central Business District, a stone’s throw from the Superdome. While it was high and dry just after Katrina blew through, it was soon surrounded by six feet of water. That meant that the building where 70 percent of Entergy’s employees reported to work every day was unreachable.  “We had to make the assumption that because of the substantial damage downtown, we would not be returning to normal operations there in a week or two,” says Johnson. He and other senior executives began to look for a temporary home for the company. They found one in the former Worldcom headquarters in Clinton, Miss., just outside Jackson.

The IT team was charged with getting the telecom and IT infrastructure in place at interim headquarters. The enterprise business continuity team, having located all employees either by phone or Internet, began to assist those who could return to work in finding temporary housing in the area.

Unlike some other disasters that Entergy has weathered, this one had a huge business continuity element to it and not just in terms of the physical headquarters. “The business continuity plan was a major part of our restoration, says Johnson. “As we located people were able to immediately start looking at how we could best mobilize our employees. We knew who was where so we could start to look at adding equipment and computers in new locations where necessary and mapping our needs to the resources we had.”

On the ninth of September, Entergy deployed an electronic survey on its intranet, asking all employees to sign in and provide updated information on their location and personal situation. “That gave us a pretty good handle of where everyone is, what their family situation is, and whether they are available to be redeployed,” Johnson says. Armed with a spreadsheet of that data, Entergy’s business continuity team began to build ad-hoc teams based on geography and skills, not job title. “Almost no one is doing the job they had before,” Johnson says. In some cases, employees are beginning to report to Entergy locations in Houston and Little Rock, for example. In other cases, workers are telecommuting.

Entergy’s ultimate goal is to return to New Orleans and its headquarters in the CBD. But a lot has to happen within the city before that can happen. “There’s a lot of cleanup to be done and restoration of the basic infrastructure there,” says Johnson. The good news has been the word that draining of the city will take less time than originally estimated.

As of Sept. 15 at 5 p.m., the company had restored electrical service to more than 860,000 the 1.1 million customers affected by Hurricane Katrina. Each day, more employees return to work. The company has created a task force called the Entergy Virtual Office team to look at what it will take to operate efficiently in Entergy’s newly distributed office environment, leveraging existing and new technologies. And of course, they are already looking for ways to do things better, if another disaster should strike.

“The sheer magnitude of this makes it a kind of one-time event and we hope this will never happen to this degree again,” says Johnson. “But you can’t go through something this significant and not find ways to do better.”

One thing that probably won’t be on that list – but is as surprising as anything else that’s happened – was just how much Johnson, his peers, and his employees could take. “I don’t think anyone ever felt overwhelmed. Even in the darkest period, no pun intended, when there was no power at the Power House, there was never a sense of panic,” Johnson says. “We had a lot of people worried about their homes and their city, but they went to work collecting facts to figure out where we should start and what we should do.”

For the next few weeks, the business continuity work continues for Johnson, who, with his wife, and two daughters and his dog, calls the Jackson Hilton home.  Most of his time, of course, is spent at work, staring at a computer screen that is adorned with a picture taken the Sunday after the storm hit. It shows a few office buildings in CBD that Entergy was able to restore power to in the early going. “You could see that little piece of the CBD lit up and reflecting off the river,” says Johnson. “It didn’t get much media coverage. But I made it my screensaver. I know it’s just one small area. But it gives you hope.”

Copyright © 2005 IDG Communications, Inc.

The 10 most powerful cybersecurity companies