Getting started with security automation

Network engineer Jose Arellano concedes that “the hardest part of my day” is keeping the network safe for 12,700 students, 1,900 staff and more than 10,000 connected devices at West Aurora School District 129 in Illinois. The two-person security team once focused primarily on getting the network running as securely and efficiently as possible for teachers and students. “We always focused on what was inside,” with the school’s limited resources and budget, Arellano says. 

When a DDoS attack took down the district’s network for more than six weeks, however, they struggled to identify the problem. Now he’s had to shift his focus from prevention-only approaches to detection and response. “It is an incredibly difficult job,” he says.

Arellano’s frustration is shared by a growing number of security professionals, and that’s partly due to the number of reported vulnerabilities each year. Threat intelligence firm Risk Based Security logged nearly 5,000 new vulnerability disclosures in the first quarter of 2020 alone. It’s hard for stretched security teams to evaluate the risk those vulnerabilities pose. 

Nearly all the respondents to the Dimensional Research 2020 State of SecOps and Automation survey reported that high alert volumes created problems for security teams; 83% said their teams experienced alert fatique. Most companies with more than 10,000 employees receive more than 1,000 alerts per day.  

The WannaCry ransomware attack marked a rise in global assaults involving malware, ransomware, phishing schemes and various strikes by bad actors — and most are indiscriminate about their targets. Many organizations, regardless of size, receive tens of thousands of security alerts from their monitoring systems every day. Some 37% of banks, for example, receive more than 200,000 security alerts a day about possible attacks, according to research firm Ovum. 

The onslaught of attacks only adds to the pain points for security teams. Not only do organizations have to sift through data and prioritize responses to thousands of alerts, but taking action requires hands-on investigating by cyber professionals who are already in short supply. Eighty-one percent of respondents to a survey conducted by Oxford Economics on behalf of ServiceNow said they were concerned about detected security breaches going unaddressed. 

A slew of new automated detection and incident response technologies are popping up to provide some relief, but many companies are still averse to security automation, says Joseph Blankenship, senior analyst serving security and risk professionals at Forrester Research. “In the past, [automation] has caused us problems,” Blankenship says. “We’ve stopped legitimate traffic, caused outages. There’s a lot of issues with taking automated action without necessarily having somebody look at the action and verify it.”

Now there might be some renewed optimism.  “Not until recently have we opened up APIs where we’ve got the ability to not only pull data out beyond just plain and simple log data, or to push an action back. There’s more sharing between platforms, and we’ve created this automation and orchestration layer thanks to APIs that allow a little more free-exchange of data,” says Blankenship.

Orchestration and automation are potential solutions, says Jon Oltsik, senior principal analyst at ESG and founder of the firm’s cybersecurity service, “but you really can only toe-dip into that. It won’t solve all of your problems. Sometimes it means changing your processes, as well.”

Companies that have used automation to deal with alert overload are seeing results, according to the Dimensional Research survey. While 34% of security teams with low levels of automation deal with most security alerts in a day, 65% report they resolve alerts in a day with automation. The majority of respondents (92%) said that automation was the best solution to deal with large volumes of alerts.

Organizations have a host of automated incident response solutions to choose from, and one size certainly does not fit all. Three organizations share their own cybersecurity challenges and response strategies.

Managing the deluge of security data

At managed care services provider CareWorks, the security data being gathered by its security tools at 88 U.S. and six international locations was proving to be too voluminous to handle, “even if we had the right staffing level in IT,” says Bart Murphy, CIO and CTO. “You have to do more with less.”

Murphy started looking for ways to gather all the data from its vulnerability scanner, security analytics software and endpoint solutions, and then automate at least some of the workflow.

CareWorks already used ServiceNow’s platform-as-a-service to automate enterprise IT operations. So in March 2017, the company added the vendor’s security operations module. While still in the early days of adoption, the company has already integrated tools like Symantec, Nessus, LogRythm and Tanium to identify workflows that we can automate. “We’ll eventually leverage orchestration to actually [respond to threats] by itself and report back,” Murphy says.

Today, the SecOps module tracks all the activity associated with a potential or real security incidents without having to manually go through myriad logs. It’s too early to tell how much time and manpower will be saved down the road.  Right now, Murphy’s goal is “to ensure that we’re as protected and preventative as possible for things that we know,” but it will take time to build confidence in security automation, he says.

“There is a level of validation that has to occur over time to get comfortable with that automation,” he says. “I don’t have unrealistic expectations about how much should be automated over a six to 12-month period. I’d rather have 10 really thought-out and tested automated [processes] than have 100 that weren’t. Make sure the team understands the goal and doesn’t automate for automation’s sake.”

Fewer security tools are better than more

When it comes to cybersecurity, Finning International CISO Suzie Smibert is all about simplification.  In terms of cyber response technology, “there are too many vendors today,” says Smibert, who is also global director enterprise architecture at the Vancouver-based firm, the world’s largest supplier of Caterpillar products and support services.

Finning receives tens of thousands of security alerts daily, made even more complicated with servers and a network covering three geographies and more than 13,000 employees across the globe who each carry more than one connected device. “Adding more security tools doesn’t increase your security. It might make it worse because managing that complex environment where you have 100 different security widgets could introduce a false sense of security,” Smibert says. What’s more, “If you have 10 devices doing only one function in cybersecurity, then you have 10 times the training and expense.”

Smibert chose only a handful of multi-function security tools to detect and respond to cyberattacks — a combined network, cloud and endpoint security platform that automates prevention against attacks, a cloud-delivered endpoint protection solution, and an analytics-driven SIEM. (She declined to identify these tools by name for fear of that she’ll receive a deluge of calls from competitors, she says.)

Her team can now decipher thousands of alerts daily and pull only those that require investigation — about 20 to 40 per day. Smibert says she’s fortunate to have enough skilled security professionals to do the manual legwork, so she not rushing into more orchestration and automation.

“I’m not comfortable yet to automate the security of the data or the function of a system that is so critical to the organization,” particularly legacy applications, she says, “but that doesn’t mean it won’t happen. “Some of these systems have not been designed for automation. If you’ve automated a false positive or created a chain reaction, that has a much more negative impact than a small and contained security incident.”

Network traffic analytics make two feel like 200

K-12 schools are typically not as well staffed or budgeted for cybersecurity as private organizations. West Aurora School District 129 turned to incident response software to help fill the gaps.

A two-person IT team manages infrastructure at 18 schools in the district. At the start of the school year in August 2016, the district’s wireless network crashed, and nobody — not even the district’s ISP — could locate the source of the problem. “We were a Cisco shop, [but] we lacked a lot of the features that would have been available through firmware updates (through Cisco’s Smartnet service), so our network visibility was very minimal,” Arellano recalls.

The ISP suggested that the school district might be a test bed for a major attack, and “it scared us,” he says.  The problem lingered for six more weeks until Arellano installed incident response software that analyzes traffic and forensic data to find the root cause of disruptions.

Using Plixer’s network traffic analytics system, Scrutinizer, Arellano immediately saw the flood of DDoS alerts. Through packet captures, he noticed a lot of DNS responses were coming out of the U.S. Consumer Products Safety Commission. “This is how we identified what kind of attack it even was,” he recalls. A DNS reflection attack allowed the hacker to spoof the school’s address and request massive amounts of records from CPSC that were being sent. The next step was to stop it.

Arellano was able to narrow down incidents by now-visible time stamps and IP addresses, and pulled only the data that related to the incident. He zeroed in on a wired classroom on the second floor of one school. “We noticed a student deleting old records. After we got the student’s ID, we dug up records and found he was using a web-stressing website, available online for about $10 a month, to launch the attacks.  Since then, two other similar attacks have been prevented.”

“The 21^st Century version of pulling a fire alarm is launching a DDoS attack,” says Don Ringelestein, director of technology. “We used to be a reactive environment, but now we’re more proactive. There are many occasions where I see problems coming up and am able to stop it before it becomes disruptive” with incident response tools, he says.

Outside security service providers can help

Many organizations that feel outgunned and understaffed by cybersecurity threats are seeking help from service providers to do the automation and orchestration for them. By 2020, Gartner predicts that 15%t of midsize and enterprise organizations will be using services like managed detection and response, up from less than 1% in 2016.

“I’m a big believer in using service providers because these are once or twice a year incidents for many companies,” says Pete Lindstrom, vice president of security strategies at IDC. “The only way to get a sense for the nature of the risk is through service providers.  We see this in Trustwave, FireEye” and about two dozen other providers, he says.

5 machine learning technologies that help with security automation

According to a survey ESG conducted last fall, two thirds of organizations consider automation of security analytics and operations to be a priority, and 39 percent have already deployed machine learning technologies to help meet that challenge. Just what are these machine learning technologies?

1. Anomaly detection

One common use of machine learning technologies is for anomaly detection. If a company has a baseline set of data about network traffic or user behaviors, then machine learning can be used to spot incidents that fall outside the norms. For example, if an employee normally works during regular business hours and logs in from their work computer, then an after-hours login from a foreign country would be unusual — and potentially malicious.

Machine learning systems are typically trained on a historical data set, then look for anything new or unusual. The training needs to be refreshed regularly, since employees, networks, and other systems change over time. However, while an anomaly detection system might report unusual events, it won’t tell you whether those events are signs of malicious activity.

2. Cluster analysis

Another common machine learning algorithm is cluster analysis. With a large set of user behavior data, for example, cluster analysis can determine that there’s a group of employees who travel a lot and have certain other behaviors in common, and another group of employees who tend to work in one location.

The clustering algorithms can look at a much larger variety of factors and behaviors than a human can and update the clusters in real time. It still usually takes a human to look at the clusters or anomalies and determine what they mean: Is there a cluster of weird behaviors because the company going in a new direction, or is there something suspicious going on?

3. Classification

Given a big enough set of data that comes pre-divided into categories, machine learning can identify which category a new piece of data belongs to. For example, if there’s a large collection of software that’s already been divided into malware and legitimate applications, a machine learning system can tell whether a previously unseen app is malicious.

As data sets get larger and algorithms get smarter, the error rates go down, making the technology increasingly useful for cybersecurity. The same technology can be applied to a wide variety of security challenges, beyond categorizing malware. For example, with enough historical data about which anomalies wound up being malicious, a classification system can be combined with an anomaly detection system to reduce the number of incidents security professionals need to deal with.

4. Predictions

The next step up in cybersecurity intelligence is to have learning systems watch as security professionals deal with incidents. What are the typical responses for particular security issues?

The challenge here is gathering enough historical data to make meaningful predictions. Every company is slightly different, and even within a company, different analysts might react to problems in different ways. However, not only are data sets getting better in this area, and algorithms getting better at deriving meaning from that data, but vendors are working to create anonymized data sets from pools of customer data. Now, a cybersecurity platform can make intelligent predictions about what the response to a particular set of incidents is likely to be, and turn that into a set of recommendations.

5. Automated remediation

At some point in the future, once a company is comfortable enough with the recommendations provided, it can start to do automatic remediation for the recommendations that carry the lowest risk to the company, or that offer the highest benefit. Getting to this step takes time — time for the systems to get smart enough to be useful and time for the company to learn to trust them.

Before a company can automate its security responses, it must have the basics in place, including an orchestration framework, security playbooks, and a process to collect security responses. Orchestration allows one security system to trigger an action in a different system, without requiring a human being to log into individual systems and manually execute commands. This is usually accomplished through the use of APIs and some kind of orchestration fabric or platform, either completely home-grown, assembled from open source components, or purchased from third-party providers.

The next step is to create playbooks — a set of steps to be carried out if a particular incident takes places. These playbooks are typically assembled manually, based on the expertise and experience of security staffers. These playbooks can immediately reduce workloads and speed up responses by automating the most common tasks, while at the same time helping the organization discover holes in its integration and orchestration framework.

In the meantime

Oltsik advises security leaders who are on the path to automated incident response to stop buying point tools until they address their own operational challenges. “Talk to your people and figure out where your biggest pain point is.  Where does it take two hours to resolve issues? Where is it difficult to get people to work together or get the data that you need for investigations or forensics?  That’s where you start to point orchestration and automation tools. These things can’t be mandates. You have to get your people on board and get everyone working in the same direction.”

When ready to automate, go for the low-hanging fruit, Oltsik says.  “If threat intelligence tells you a particular IP address or web domain is bad and it gives you an 80 percent confidence rating that it’s bad — you shouldn’t have to get a person in the middle of that.”

The next step, orchestration, takes time, Oltsik says. It assumes you either have a security process in place, or you’ve take the time to go through all the tasks associated with the process, and you know how to apply technology “to make that [response] better,” Oltsik says. “That may take a while.”

It’s also important to have a lot of review cycles for any new automation or orchestration processes, he says. “What did you miss that you shouldn’t have? What could you do better next time? Did the process flow like it should have, or should there have been extra steps or missing steps?”

Smibert believes the road to broad adoption of incident response automation will be similar to the path taken toward cloud adoption. “Five to 10 years ago everyone was scared of the cloud, but the industry has proven that when you have a strategic and thoughtful approach in embracing a cloud technology, you can do wonders. I believe the same holds true for security automation. Once the industry agrees, and we have early adopters that have great success, then we’ll get more adoption, and with more adoption will come more innovation. Then, potentially, we’ll see security automation as popular as cloud is today.”