Security Simulations: This Is Only A Test

Imagine your workweek unfolding like this:

[Day One] A smoke alarm sounds at corporate headquarters on the second floor. Maintenance tells you that the detector (not a standard company-issued brand) is plugged into a network cable at your financial services company.

[Day Two] Traffic on your corporate network spikes to 20 percent above normalmost of it coming from business partners cross-selling your company’s products.

[Day Four] A security guard, reviewing the past week’s security tapes, notices a janitor escorting a stranger into the office via a side door. This person traveled to the second floor, alone, stayed for two hours, then left via the same side door.

[Day Five] The financial services Information Sharing and Analysis Center (FS-ISAC) releases a bulletin about a new vulnerability in various versions of OpenSSL (a Web security toolkit) that can cause a denial-of-service condition. Hours later, your HR department can’t access the company’s personnel system, including payroll.

These occurrences are all related, and could indicate not only trouble for your company, but indeed a coming infrastructure breakdown. But how would you recognize this? How would the maintenance tech and security guard know to contact the right people about the network jack and the visiting stranger? And even if they did, what do you do about it?

The best way to prepare for the worst is through practicewhich is why use of tabletop exercises is on the rise.

The scenario described here is one of several played out by 150 security executives from the financial sector in a two-day emergency response exercise hosted by the FS-ISAC at the Don CeSar Hotel in St. Pete Beach, Fla., in April 2004. Since March 2003, five such convocations (at a cost of $250,000 each) have been hosted by the Department of Homeland Security or the Secret Service for the financial, IT, and oil and gas industries.

“These scenarios hone your skills and remind you that your crisis plan needs to be reviewed and updated regularly and that threats always change,” says Ron Hicks, manager of corporate security for Anadarko Petroleum. Hicks participated in an oil and gas industry tabletop in Houston in February, which was attended by about 70 percent corporate and 30 percent IT security executives. The blending of online and physical attacks is one of the newest wrinkles in the simulation game, but the bottom line for security pros of all sorts is that red team, blue team exercises can help create more detailed and useful incident response plans.Let the Games BeginThe tabletops run by the Secret Service-based Electronic Crimes Task Force (ECTF) are an extension of its charter to foster coordination between private-sector companies. Since 1998, the ECTF has held quarterly meetings with security executives from critical infrastructure companies, whom they also assist with cybercrime investigations. The exercises are developed and officiated by the Guidry Group, a corporate security firm that, for the past decade, has worked with the Secret Service and other law enforcement agencies in cases involving corporate espionage and fraud.

At the financial services tabletop in April, scenarios played out in two “situation rooms” with up to 15 participants in each. The people in each situation room acted as employees for two financial companies that share customers in a longstanding business relationship. Events in the situation rooms start out simplya cyberprotest, an increase in intrusion detection alarms, a burglary and unauthorized software discovered on a server. At first, these events seem unrelated, and contact between affected parties is minimal.

Outside the situation rooms, the other attendees watch events unfold over two big screens. Between sessions, moderators roam the audience with microphones taking feedback. And during the sessions, audience members use wireless handhelds to answer multiple-choice questions posted on a third screen. For example, when the exercises heated up and it became clear the authorities would be needed, the moderator posted the question: “Whom would you contact about this behavior/instance? 1. FBI; 2. Secret Service; 3. ISAC; 4. DHS; 5. No one; 6. Other companies in your sector.” Most attendees didn’t know whom to contact, according to Jared Graves, director of the tabletop business unit at Guidry.

“The answer is different for each sector. Financial firms contact the Secret Service because it’s part of the Treasury Department. Oil and gas folks call the FBI because the pipeline’s a target for terrorists. And so on,” he says.

The scenarios change every hour or so, at which time new participants go into the situation rooms, old participants move to the audience, and situations continue to escalate until the two-day exercise culminates in a wide-scale attack on the infrastructure. Then the moderators capture final attendee input. The last two hours are dedicated to sharing the findings and conclusions.

Hicks, who is a former secret service agent, participated in a situation room about midway through the enactments at the oil and gas tabletop earlier this year.

“It was intense,” he says. “Events change every 10 minutes while you’re determining damage, injuries, threat to the larger public, containment and restoration. And it’s not only how you handle incidents, it’s how incidents roll over to companies and affect them.”

The overarching findings, according to the organizers, are that communication is critical in making informed decisions, and that every company needs an action plan and contact list that are both rehearsed and up-to-date. But the devil is in the details. To make the exercises as valuable as possible, the Guidry Group contracted the development of the scenarios to a team of experts from the Center for Infrastructure and Security at the University of Texas at San Antonio that polled industry leaders about their chief security issues. Then they based most of the scenarios on real-world events the respondents had experienced.

“The goal was to represent the incidents as things that really happened, or were highly possible, to make them realistic. For the most part, the tabletops pulled that off,” says Eric Guerrino, senior vice president of information security at The Bank of New York, who participated in the April FS-ISAC exercise. “For example, we’re more aware these days that worms and viruses can take down certain segments of the infrastructure. So malware was used in some of the scenarios.”

The April tabletop is Guerrino’s second tabletop exercise. He attended a similar exercise held in New York a year earlier. And, while he felt there was some confusion and inconsistency in the changing of players and escalating scenarios, he also believes both tabletops helped him look at his vulnerabilities and response plan in a different way.

“One of the things you learn quickly is you have to identify ahead of time your points of contact if something does happen,” he explains. “And you learn you’re not operating in a vacuumthat you need help from your physical security department, from various public agencies, your business partners and your industry at large in some cases.”Communication BreakdownFailure to communicate can even jeopardize the business relationships. At the financial services tabletop in April, the two fictitious financial firmsdubbed Firms A and Bstarted out as partners. Firm B was hammered with cyberattacks coming from Firm A; however, following corporate legal advice, Firm A wouldn’t admit it had a problem until it could confirm exactly what that problem was.

“There were a number of opportunities for the two fictitious firms to share information. As expected, they didn’t. And by the time they figured out the two events were related, it was too late,” says Byron Yancey, executive director of the FS-ISAC. “It got hard to tell if the actors were role-playing or serious. They got angry. At the end, they were moving toward terminating the business relationship. The lesson here is that if you have a trust relationship and an open channel, you can minimize your risk by sharing pertinent information.”

Another point the exercises drove home: Anything can be related to anything, and customers, suppliers and infrastructure companies are inextricably linked. At the oil and gas tabletop games, cyber and physical attacks nearly took down the entire supply chain.

“During my time in the situation room, we had an unauthorized vehicle located inside the premises, which followed the initial suspicion that we were already under cyberattack and cyberdistortion,” says Everett Teglas, regional manager for ChevronTexaco’s Global Security team. “As it normally does in a crisis, information came in sporadic spurts from all sourcesa port, the endpoint customer, the transportation company and the storage facility. Because they were able to coordinate this information, participants realized the attack was designed to cripple the supply chain of the oil and gas infrastructure.”

Attendees say they plan to practice more of these types of exercises during future events. And they’ll also be expanding their own response rehearsals with more blended threats across more corporate departments and even with business partners.

“This exercise drove home for me how cyberattacks can be a mechanism to set off a chain reaction incorporating all measures of a security infrastructure,” Teglas adds.Connect the DotsSometimes incidents are bigger than just one industry. For example, the Northeast blackout of 2003 called for cross-sector communications because all businesses require electricity. Another example: It takes oil to create power. Yet, without water, you can’t pump oilanother finding from the Houston oil and gas tabletop in February.

This is why FS-ISAC chair Suzanne Gorman, who also heads the ISAC Council with membership from all sectors, planned a cross-sector tabletop at the ISAC Council’s annual meeting in Florida in October. In future events, Gorman aims to include a larger presence of physical security leadership than the 15 percent representation in the April financial services tabletop. She also hopes to see more CEOs there. The ECTF made a concerted effort to bring infrastructure CEOs into the tabletops. While the majority of the invited CEOs instead sent their executive security teams to the exercises, about a half-dozen CEOs attended each tabletop, according to Graves.

So while today’s simulations are giving CSOs a detailed look at preventing disaster, tomorrow’s events may provide even greater value by helping other business leaders connect the dots as well.