Be wary of vendors touting superior data science

In the highly competitive market for security tools, many vendors make the misleading claim of having the best of everything, and at this point in time “everything” often refers to data science, machine learning and AI. The result is an arms race of claims about tools that  “automagically” address security problems, according to Forrester Research.

In its recent report “The Top Security Technology Trends to Watch, 2017,” the analyst firm called out the battle of the data science algorithms, saying, “When virtually every security vendor makes the claim that they’re using artificial intelligence or machine learning for detection, security decision makers are left shaking their heads, trying to figure out what’s real and what’s not.”

Still, decision makers need solutions. And it should be noted that, “data science has been part of cybersecurity for as long as there has been a category called cybersecurity. Machine learning and artificial intelligence do have roles to play in security, but they are not a panacea for the prevention of all cyberattacks,” the report said.

As is the case with many tools from antivirus to firewalls, data science is useful and has its place in the overall security ecosystem, but it should be used to inform “decision making as a supplement to rules-based or signature-based detection,” the report said.

Stephen Pieraldi, senior director of AI and machine learning strategy, at E8 Security said the very question ‘What is artificial intelligence and how does it help me?’ leads to the answer.

“It’s artificial,” Pieraldi said. There are a variety of data science tool kits, but even with many different tools pulled together to create the widely known IBM Watson, it’s limited in its capabilities. “Watson can win at Jeopardy, but if you try to play Family Feud, Watson can’t win. It’s powerful in the right context,” Pieraldi said.

Because the industry is cursed with a deficit of people, security teams need to rely on algorithms that can detect more powerful attacks. “Humans doing the work can’t keep up, and the answer to that is automation. They need to have some form of machine learning,” said Pieraldi. 

The challenge today is to find the data scientists who aren’t obsessed with the data, said Pieraldi, but are obsessed with the tools they can create. As the security industry continues to evolve, there will continue to be fertile ground with people developing solutions. “Eventually that grows up into a full industry solution or a consolidation of a component of all the tools,” Pieraldi said.

In these early stages of the evolution, they are still fighting because, Pieraldi said, “The best solutions haven’t been discovered yet.”

One thing CISOs need to appreciate is not whether the hype is right or wrong. “Hype will exist until it is right or wrong,” said Peraldi, “accept the hype and work through the challenges.”

Joey Peloquin, director of cloud security operations for Citrix, said that the industry is at a point where vendors believe that they can’t be taken seriously unless they are inserting the buzzwords of AI and machine learning.

“It really isn’t my algorithm is better than yours, though. Machine learning and AI are a little different, and overhyping is doing a disservice,” Peloquin said.

More to the point, it’s important to examine the elements of a data science strategy and understand why we are interested in these tools. “We have an immense amount of information today that human beings can’t analyze in a meaningful way,” Peloquin said.

Intelligence is overhyped, potentially because of its sundry definitions across both the public and private sector. “At the end of the day, it’s about intelligence. What data science is about is being able to leverage the huge amount of information we have, and to analyze it, enrich it, and make it actionable in a proactive instead of a reactive way,” Peloquin said.

When developing a particular tool, data science informs which path to go down said Peloquin. “Vendors that are overhyping don’t understand the problem we are trying to solve.”

So, how do you know whether their algorithm is doing what it’s supposed to do? 

In order to make informed decisions, CSOs should ask vendors questions like, Do you have a Phd data scientist on staff? Who leads your team? Where are they from? What is their background and experience?

“They [CSOs] need to be smart enough to ask the vendor to ensure that their products are not just marketing speak. If they [the vendor] based all of their capabilities on the output of automated tools rather than experts in the field that can do targeted attacks, then I would argue that their solution is not as mature as they are claiming it to be,” Peloquin said.

There are products that make the claim that they have blocked 100% of their attacks with no false negatives, but Peloquin said it’s important to ask whether the pen testing has been done by an actual person who knows how to bypass controls and customize payloads.

Then, said Peloquin, go and research those individuals. “Don’t take the responses from a vendor at face value.”

Data science is changing the game, said Peloquin. “I’m excited about how it is enabling us in the security field to do so much more so that we are not wasting money on solutions that are not going to have a major impact.”