Ask me anything: Insight Engines lets you talk your network into revealing threats

In the past, CSO has looked at solutions to help automate security processes, or even ways to outsource increasingly critical security functions like threat hunting. But those all come with risks, and as intelligent as artificial intelligence is getting, it’s still nowhere near as good as a knowledgeable human at following hunches and seeking out trouble.

The Insight Engines program is an attempt to leverage the power of computers to do the heavy cybersecurity lifting, while keeping humans solidly in charge of each investigation. And, it does this with a completely on-premises solution that is not vulnerable to snooping, or placed at the mercy of outside connectivity issues.

At its heart, the Insight Engines program is a natural language processor that is tied to an incredibly complex backend programming interface. The goal is to have cybersecurity team members of any skill level ask questions of Insight Engines in plain language, and then let the program put together an advanced query leveraging all available data pools to return an answer.

Right now, the Insight Engines suite only runs under Splunk, though that does give it access to every security tool that is already integrated into the popular Security Information and Event Management (SIEM) program. It is installed as a Splunk application, ready within about 30 minutes of activation for existing Splunk users. Thereafter it needs about a day to examine all connected data, though the program keeps learning after it’s in place, so it eventually becomes completely in sync with the network that it’s protecting. There is also a one to two-week process where Insight Engines will help new customers clean up and refine their datasets in order to get the most out of their new security tool.

At a high level, one can almost think of the Insight Engines tool as Google for network security, allowing natural language searches and returning honed information to answer each query. This does not do the program justice at all, but is a good starting point for understanding how it works.

John Breeden II/IDG

A typical session with Insight Engines involves an analyst asking the program a question. In our test, we started with something simple like, “Are there any hosts on our test network that communicated with China this month?” Insight Engines returned a positive answer, with supporting materials showing what file types or data was exchanged, which hosts, clients and users were involved, and if any of that activity generated an alert from any of the existing security tools installed on the network. It even provided us with a map of all the locations in China where data packets were being sent.

John Breeden II/IDG

When we moved to a more advanced query, the program didn’t falter. Our next question, “Have any users failed 500 or more login attempts before succeeding?” returned a few responses, including a superuser, Bob Smith. From there, we could hone our query along the lines of, “After User Bob Smith successfully accessed the network yesterday, what services and data did he access?” We could even ask complex questions such as comparing Bob Smith’s network usage on Friday (after the 500 failed logins) with his historical patterns. This kind of query and line of thought is gold for a network threat hunter, yet it’s almost never used because low level analysts don’t know how to write queries to get those answers, and experienced threat hunters need to put in hours, or sometimes days, writing scripts and then refining them. It’s no wonder that threats can take months to uncover.

John Breeden II/IDG

Insight Engines handles the hard work on the backend. In fact, every query that it creates is an accelerated Splunk request designed to quickly return results. Very few Splunk users know how to craft an advanced, accelerated query, and even if they do, it takes time – and the entire process can be upended by small typos or other tiny mistakes.

The Insight Engines program created accelerated queries for us in just a few seconds, and allowed us to examine the created strings. Had we been expert in making accelerated queries, we could have tweaked them to display different information. However, using Insight Engines’ natural language option, typing in our queries like having a conversation with the program, seemed just as good.

John Breeden II/IDG

In this way, the Insight Engines program attacks both ends of the cybersecurity manpower problem. For less-skilled workers, it allows them to use their insight to hunt and uncover threats without needing to know how to program Splunk or getting a data science degree. In fact, with a little bit of training, companies could bring in professionals like retired police officers who know how to think like a criminal, or who are predisposed to following good investigative techniques. They could conduct threat hunting using their intuition regardless of their programming skill, or lack thereof, and it probably would not take long before they became valued members of the cybersecurity team.

At the high-end, top professionals could use the Insight Engines program to quickly craft accelerated Splunk queries, saving hours of time each day and enabling them to quickly access threats to determine if there is a danger to the network. They can even use the program to set up daily queries to help automate the security process. For example, if they found our question about finding any user who failed a set number of logins and suddenly got network access to be valuable, Insight Engines could be set to re-ask that same question automatically every day, and alert if anything is detected.

John Breeden II/IDG

It’s unfortunate that the Insight Engines program only runs under Splunk for now, though other versions may one day be forthcoming. For any organization that has already installed Splunk, there is no reason not to also deploy the Insight Engines program. It will assist analysts of all skill levels, and even possibly help fill vacant SIEM jobs with insightful investigators that may not have programming experience or deep IT skills. So long as users have a basic understanding of how a network operates, identifying potential threats with Insight Engines should be an achievable goal. For the high-end analysts, it might end up being little more than a time saver, but a good one in a position where every second counts.