Cybersecurity operations: Don’t wait for the alert

One of the reasons that we in the cybersecurity industry continue to lose to bad actors is our lack of pro-activity. While many organizations have made big investments in their security functions, they tend focus largely on reacting to alerts and responding to known situations.  This is a bit like arming your burglar alarm after the robbery.

For the typical enterprise, the key aspect of the cybersecurity function involves a security operations center (SOC), typically a large room with big monitors lining the walls. The SOC Analysts watch the monitors for alerts, and when they find one, they follow a playbook that defines the corporate standard for responding to that type of alert. Such a system is well designed as a reactive approach to security.  They are popular, in part, because of their scalability. They are staffed by folks that in most cases have limited experience and undeveloped investigatory skills. These Analysts are often comparably inexpensive, and if activity warrants, these operations can be easily enlarged. The SOC would seem to be the ideal solution keeping up with our abundant cybersecurity activities. Unfortunately, a SOC is usually only good at responding to alerts, with the damage often being done before the first alert sounds.

A recent published report in Crowdstrike indicated that the average dwell time for a network intrusion, from entry to discovery, was 229 days. As such, a bad actor has plenty of time to steal data or damage systems before the first alert is ever sounded. A SOC responding to alerts will provide little benefit in responding to such an attack and can only serve to minimize and quantify the exposure.

The SOC does have its place. My organization uses a commercial SOC service from a well know company. They generate many alerts, each of which is investigated by my in-house team. Their alerts are often for conditions my team would not have the time to find. Unfortunately, we generally do not get such an alert until well after the problem has occurred, and they are often for lower priority issues. These alerts do provide value, because they allow us to clean up infections, and tighten our preventive measures. They would not, however, prevent damage from a significant incident. Thus, if we try to rely on the SOC as our primary means of enforcing cybersecurity, we are doomed to failure.

So, if the SOC is not the best answer, what is?  I would suggest that the best approach is a team of folks with strong investigative skills, who spend their days finding issues before the first alert sounds. These individuals are far different in skill and mindsets from the typical SOC occupant. They have an instinct to follow their noses, and a strong drive to find an explanation for every anomaly they identify. They usually possess the tenacity to not give up on an issue until they have an answer.

True, these individuals are more expensive, and much harder to find. When you do find them however and empower them to dig for issues rather than waiting on them to surface via an alert, they quickly pay for themselves.

Here are some tips for achieving a true investigative function:

Hire the right folks

This almost goes without saying, but having a team with investigative skills and mindset is critical to achieving an effective organization. I prioritize investigative abilities over actual experience in most cases, to find the people I need. Selection requires a careful interview process, because these special skills will often not jump out at the reader from a resume page.

Provide the needed tools

Although you don’t need to spend a fortune on tools for a good team, certain fundamental systems are essential for a good investigative function. The cornerstone for this is the Security Incident Event Management system (SIEM), which collects log records from various systems into a single repository. This requirement does not stop at the purchase of the system, however. It is critical that key servers, network devices, and even workstations be setup to send their log records to the SIEM. A SIEM with the necessary data allows an investigator to correlate events from various logs to look for behavioral patterns. Without an SIEM, the Analyst would need to look at too many individual logs, hampering the process.

Provide good training

Ongoing training is critical to a good investigative function. Sadly, I have seen few good training opportunities specific to this need. In my experience, training designed for red teams, which many companies use to test security from the outside of a company in, works best for investigators, given that it allows the Analysts to put themselves in the position of an attacker, so they have a better idea what to look for.

Give them time

A good investigator must have time to follow their gut. If they see something that bothers them, they must be allowed to dig into the issue until they find a problem or satisfy themselves that all is well. In the process, they will go down many rabbit holes and find nothing of note, but each one is an educational experience. Rather than scheduling them for too many routine activities, free them to do what they do best—dig.

Bottom line: An SOC is a useful part of our cybersecurity arsenal, but its main benefit will be in helping to minimize damage from an issue that has already happened. A strong investigative team, on the other hand, can help to identify and resolve issues before they cause major damage, which is always our preference.