How San Diego fights off 500,000 cyberattacks a day

Nearly 27 years of network and cybersecurity experience with the Department of Defense didn’t prepare Gary Hayslip for the collection of disparate technologies he encountered when he joined the city of San Diego.

“Cities don’t get rid of anything. If it works, why get rid of it? So you end up having a lot of diverse technologies connected together. You may have something that’s 15 years old connected to stuff that’s state of the art,” says Hayslip, whose DoD tenure included 20 years of active-duty military service and seven years working in civil service for the military.

“Police cars, ambulances, libraries, water treatment facilities, golf courses … One of the things you learn real quickly: the city of San Diego is $4 billion business. And cities don’t shut down. They run 24/7,” he says. “My almost 27 years in DoD did not prepare me for how interesting city networks are.”

As San Diego has explored smart city initiatives to improve areas such as public safety and transportation, the introduction of Internet-connected devices has only ramped up the complexity.

“Being a smart city, where you’re doing a lot of industrialized IoT, and you’re bringing in even more of these intelligent infrastructure projects, you really start wondering – what happens if I do an update or a software change on this? Is there going to be a rolling effect? Sometimes you don’t know until you actually have issues,” Hayslip says. “As a security professional it scares me because some of it, from a risk perspective, is unknown.”

Not knowing led San Diego to get serious about improving security visibility and reducing threat exposure across its citywide IT environment, which spans 24 networks, serves more than 11,000 employees across 40 departments, and encompasses more than 40,000 end points.

“I lacked visibility into how the networks were being used, where data was going, what connections I had, what was actually out there on the network,” says Hayslip, who was hired three years ago as deputy director and the first CISO for the city.

It wasn’t a problem that could be solved overnight. Hayslip and his team came up with a five-year plan, which started with the adoption of a policy framework for computer security guidance from the National Institute of Standards and Technology (NIST). “You want to use a framework that at least gives you some building blocks to assess how bad it is, so you don’t go screaming, running into the hills. You can use this framework to figure out, maturity wise, where you’re at. Once you have that baseline, you can start putting together projects to move forward.”

The tricky part of establishing a baseline, however, is that the city is constantly changing as departments deploy new technologies and upgrade existing infrastructure. “When there’s a high rate of technology change going on in the organization, you realize very quickly that you don’t have a stable baseline and that you’re not going to have a stable baseline for quite a long time,” he says. 

With that kind of volatility in mind, Hayslip adopted NIST’s model of continuous monitoring, scanning and remediation. “It’s a whole lifecycle of how to do cybersecurity. Basically what it means is that you don’t stop. You just keep pushing forward. It involves looking at cyber as a living lifecycle of inventory and assessment, scanning, monitoring and remediation. You just do it over and over again.”

To enable this cycle, Hayslip advocated for a vulnerability scanner he worked with during his time with the military. Tenable Network Security uses a mix of sensor options, such as agent scanning, passive listening, traditional scanning and log analysis, to provide a unified view of a company’s overall risk environment. The city uses Tenable to run vulnerability assessments, inventory network traffic, identify misconfigured devices, and find missing or outdated patches.

Having that continuous visibility across a distributed enterprise network is critical given how often San Diego is targeted by cyber attackers. “We’re averaging about 500,000-plus cyberattacks a day,” Hayslip says. “A lot of them are automated, however some of them are specific; you can tell it isn’t just a machine firing off a tool. It’s specific groups who are doing the targeting.”

San Diego isn’t alone. Over the last several years, public-sector organizations have become increasing attractive targets for cybercriminals. Government is among the top five most cyber-attacked industries, according to the 2016 IBM X-Force Cyber Security Intelligence Index.

“Criminal organizations – they go where the data is at. They’re an equal opportunity exploiter, and cities tend to be very big with a lot of assets and a lot of data. So, like it or not, we’re a target,” Hayslip says.

A coordinated defense

Tenable is one of many security tools the city uses for its defenses. Tenable is integrated with Carbon Black, which provides endpoint security on the desktop, for example. “A lot of times we’ll marry Tenable with Carbon Black and we’ll get way down into the weeds on an actual desktop and find out if something is a false positive or not,” Hayslip says.

On the perimeter, Tenable is tied to PacketSled, which handles network monitoring and threat detection.

For data governance, San Diego uses technology from Varonis. “We’re using Varonis to look at where data is at, to see who’s accessing what at a data level. Then we use Tenable to verify what we’re seeing, to verify movement of assets on the network,” Hayslip says. “There are a lot of different things you can use it for.”

For unified threat monitoring, San Diego currently uses Cyphort, a platform that enables the city to see attacks in real time and see which assets in the security suite are responding to the threat. Plus, security event data from these assets and other sources gets pumped into Sumo Logic, a cloud-based analytics service for log and metrics management.

“We’re building our dashboards so we can have this one pane of glass to look at our analytics. I’m looking for shifts. I’m looking for trends in how our data is being used, how our network and assets are being used. We’re constantly using it to look for vulnerabilities,” Hayslip says.

Once vulnerabilities are spotted, the team uses AttackIQ, a live remote testing platform, to verify their security assumptions. “Attack IQ will verify if what the scans are showing is true, and if so, we use those scans to submit tickets to get things remediated,” he says.

As the city has gotten more familiar with Tenable, it’s finding more ways to use the technology. “When I originally purchased it, I thought we were only going to be using it for one thing,” Hayslip says. “Now we’re constantly looking at what else we can use it for. Tenable has become one of the core components that our whole suite revolves around.”

So far, the approach is paying dividends. Since deploying Tenable, the city went from an average of 200 infected machines per month, which was costing roughly $600 a machine in lost productivity, to an average of 35 per month. That’s saving San Diego more than $1.3 million per year in lost productivity, Hayslip estimates.

Speak the language

Hayslip has worked hard to develop relationships with city departments, which helps ensure that the security team is involved from the beginning stages of projects. He puts in the time to get to know people, how they do business, what apps and data they need, and who their customers are.

“I’m considered a partner. We help with projects at the beginning stages instead of at the end,” he says. “I’m able to help out before we even start spending any taxpayer dollars.”

But it wasn’t always that way. Early on, the security team might have been caught off guard by unfamiliar data types. “You can be running a security scan, and all of a sudden you see something, and you think you’ve got a breach or you think you’ve got a machine that’s got an infection, then you come to find out that’s not a machine at all, that’s a streetlight.”

Hayslip found he needed to rethink how he communicates security risks to the city’s departments. “I had to make some major changes, because I realized that the risk I was seeing with my scans and everything – that risk wasn’t mine. That risk was my business units’ risk. And their priorities were totally different from mine.”

A strategy that works for Hayslip is to talk about business risks as opposed to talking about cyber threats. Department personnel care about business operations, revenue streams, and providing services to citizens. “When you start talking about loss of services – and the city golf course makes $40 million a year on credit card payments – and you’re talking about impacting that, then they sit up,” he says. “You flip it so you’re not talking cyber, you’re talking business.”

When Hayslip can articulate risks from a business context, the departments can then weigh in on what’s important. “That’s why I started calling it ‘cyber as a service.’ Because they tell me what’s a priority. I can show them all the risk, I can explain the impact, how it’s going to hurt the business operations, and then we together, as partners, can decide what is a priority.”

“I still track the number of attacks, and the number of remediations, and a lot of the security issues, because that is my job, and I need to be able to show the value of what my teams are doing from a budget perspective,” Hayslip says. “But that isn’t what I would share with my business units because they don’t care about that.”

Hayslip takes his message on the road, too, to educate businesses about the importance of cybersecurity. He visits small business forums and professional groups to lend his expertise to private-sector companies that are struggling to improve their security postures. “I think it’s part of my mission, representing the city,” he says.

Looking ahead

For the long term, the goal for San Diego is to develop an infrastructure that’s flexible and resilient enough to withstand attacks and handle changes. That’s something all organizations aspire to, Hayslip says, and the city of San Diego is making progress. “Maturity-wise, we’re getting to a stage where we’re actively planning for it and building it in now,” he says.

Meanwhile, city personnel are constantly looking at new technologies to deploy – and they’re asking for more SIEM data so they can be better informed. “They really want to understand the risk to their projects and the risk to their services,” Hayslip says.

“It’s a pretty cool time to be working here at the city.”