Fear and loathing defending ICS security at DoE's CyberForce Competition

CSO goes gonzo to defend critical infrastructure from hackers as part of a cyber defense competition.

1 2 Page 2
Page 2 of 2

Seventy teams times six players is 420 competitors. Add a couple hundred green teamers playing industrial users plus the red teams and you're well under a thousand users total. The infrastructure couldn't handle all of us trying to access the sCOARboard at the same time, leading to frequent Slack messages to participants to stop all refreshing the sCOARboard at the same time.

The system was down, stayed down, and flickered on and off throughout the day.

"But our incident reports. The anomalies. How are we supposed to compete?"

"I don't know, man. I don't know."

Blood pressures rose. Fine for the undergrads but not great when your average age is 42. Let's not have a heart attack over a simulated hacking scenario, I thought.

"I've got a scaling challenge for you," I joked afterwards. "I want you to scale to three-digit users"—three fingers extended on one hand, thumb and pinky pinched together—"and I want three nines uptime." Three fingers on the other. "Three nines. Can. You. Do. It?"

"Looks like a gang sign," our mentor chuckled, mimicking my double three-finger salute.

Imagine playing baseball, but you don't know how many bases there are, how many strikes make an out, or even whether you are playing with a ball or a hand grenade. What are the rules? Where is the chalk? What is foul and what is fair? Oh, and the lights go out at random intervals and you play in the dark for a while.

Playing CyberForce to win—that's what it felt like.

While it's true the only rule in nation-state hacking is that there are no rules, and defenders in SOC chairs working at oil refineries don't have the luxury of a referee—at least, until the Geneva Convention gets a much-needed upgrade—if you're going to gamify learning, it's best if the game has, you know, like, rules, and those rules are clear and consistent and enforced equally for everybody.

"You think this is part of their research project?" I asked. "Make us squirm, see how we react?"

"No." Daren sat back in his chair, tugged at his grey goatee. "I just think they have no idea what they're doing."

"Incompetence, not malevolence."

"Yeah."

Nothing beyond our egos was riding on the results of the competition, except perhaps Little League bragging rights—and let's be clear, we were playing for Little League stakes—but I had flown out to Berkeley from New York for the competition, and many of the others had traveled long distances within the vast Golden State to be here. It was a point of pride, too. We were the greybeards of CyberForce. Were we really going to let these 19-year-olds beat us?

Two hands tied behind our backs

The backdoors were everywhere.

The Azure virtual machines (VMs) Cyber Force gave us were riddled with pre-installed rootkits and trojans and backdoored binaries, oh my, not to mention unnecessary software and services to purge with extreme prejudice. On some Linux machines, the /usr/sbin/nologin binary had been replaced with /bin/bash, thus giving password-free shell to accounts configured to have no remote access. We found that one before the competition, but what else was there? Did we find them all? Did we get everything? Would only take one oversight and we were screwed. Red team would go down their exploit list and try them all. What if we missed something?

The piranha chewed.

Plus our hands were tied. We weren't allowed to block attacking IP ranges. Doing so would render game day a moot point. If red team couldn't access our infrastructure there'd be no game to play—and in real life an attacker would simply come at you from a different IP range. A fair constraint, but still frustrating.

Nor were we allowed to use certificate-based encryption to protect sensitive data on the wire, even though we were required to ensure uninterrupted HTTP, FTP and SMTP service.

The joker of the group—that would be me, if you hadn't figured it out yet--suggested pulling a Kobayashi Maru. If we pulled it off, we would be legends. If we got caught, we'd be physically expelled from the facility. In retrospect, things were so chaotic we probably could have gotten away with it.

cyberforce 7 Karel Baloun

CTF is over. Time for beer.

Easier to cheat than to solve the real problem: How do you do the impossible? How do you secure the unsecurable?

The fear sputtered out and the loathing kicked in around 2 p.m. Loathing, and peace. I just wanted it to be over, I didn't care if we won, the whole game was chaos, if not out-right a rigged experiment, and all I wanted was a beer or three and a few laughs with my friends before heading home to New York and a long, drowsy slumber in Hell's Kitchen, Manhattan, with the relaxing sound of sirens and car horns outside my window.

We were halfway down the national rankings in the early afternoon, but somehow managed to claw our way back to the top ten before the competition ended. We didn't win CyberForce. Not this year. The UC Berkeley MICS team placed ninth out of 70 nationwide. We didn't even win the local award—the UC Davis team, led by a super smart kid who looked like he didn't have to shave very often, came in fifth nationwide.

cyberforce 8 Karel Baloun

Final results: UC Berkeley came in ninth of 70 teams nationwide.

And that early morning attack? Turned out when CyberForce had rebooted our Pis before the competition, the VNC service didn't auto-start, the way we had configured it the night before.

"Paranoia was our greatest adversary in the early going," Daren Slacked me a week later.

Risk is overrated; ruin is underrated

The day after the competition, I sat under the plane trees at the base of the Berkeley Campanile, drinking black coffee, and basking in the glorious early December sun. A surprise bell tower serenade broke out. My phone sat untouched in my pocket. The idea of plugging my brain once more into the internet.... how can I describe it? Fear and loathing, revulsion, horror. After what we'd been through the day before, just the idea of making myself vulnerable again, simply by turning on my phone, was more than I could bear.

Volunteering to be prey—on purpose, for "fun"—condensed all that emotion into one snarling day of unsuccessful flight, dodging shadows one second too late.

We are all of us now prey, both alone and together, a morsel for those predators—who now live next door to us in the shrinking global village we call home—who seek geopolitical leverage by owning critical infrastructure so unwisely connected to the internet, sabotage halfway around the world a keyboard's tap-tap away.

In cybersecurity we talk about mitigating risk, but rarely about mitigating ruin. Because ruin cannot be mitigated, only prevented, and some failure modes are so unacceptable that they must not be tolerated.

"We can defend this 98 percent of the time!" Karel told me jubilantly at the end of the day.

"What about the other 2 percent of the time?"

"What about then?"

What about then?

I sighed. A cool breeze brushed my cheek. The bell tower fell silent. I reached for my phone.

1 2 Page 2
Page 2 of 2
Get the best of CSO ... delivered. Sign up for our FREE email newsletters!