• United States




Cybersecurity operations: Don’t wait for the alert

Jul 16, 20185 mins
Application SecurityCloud SecurityData and Information Security

An SOC is a useful part of our cybersecurity arsenal, but its main benefit will be in helping to minimize damage from an issue that has already happened. A strong investigative team, on the other hand, can help to identify and resolve issues before they cause major damage, which is always our preference.

eye binary IDS Intrusion Detection System
Credit: Thinkstock

One of the reasons that we in the cybersecurity industry continue to lose to bad actors is our lack of pro-activity. While many organizations have made big investments in their security functions, they tend focus largely on reacting to alerts and responding to known situations.  This is a bit like arming your burglar alarm after the robbery.

For the typical enterprise, the key aspect of the cybersecurity function involves a security operations center (SOC), typically a large room with big monitors lining the walls. The SOC Analysts watch the monitors for alerts, and when they find one, they follow a playbook that defines the corporate standard for responding to that type of alert. Such a system is well designed as a reactive approach to security.  They are popular, in part, because of their scalability. They are staffed by folks that in most cases have limited experience and undeveloped investigatory skills. These Analysts are often comparably inexpensive, and if activity warrants, these operations can be easily enlarged. The SOC would seem to be the ideal solution keeping up with our abundant cybersecurity activities. Unfortunately, a SOC is usually only good at responding to alerts, with the damage often being done before the first alert sounds.

A recent published report in Crowdstrike indicated that the average dwell time for a network intrusion, from entry to discovery, was 229 days. As such, a bad actor has plenty of time to steal data or damage systems before the first alert is ever sounded. A SOC responding to alerts will provide little benefit in responding to such an attack and can only serve to minimize and quantify the exposure.

The SOC does have its place. My organization uses a commercial SOC service from a well know company. They generate many alerts, each of which is investigated by my in-house team. Their alerts are often for conditions my team would not have the time to find. Unfortunately, we generally do not get such an alert until well after the problem has occurred, and they are often for lower priority issues. These alerts do provide value, because they allow us to clean up infections, and tighten our preventive measures. They would not, however, prevent damage from a significant incident. Thus, if we try to rely on the SOC as our primary means of enforcing cybersecurity, we are doomed to failure.

So, if the SOC is not the best answer, what is?  I would suggest that the best approach is a team of folks with strong investigative skills, who spend their days finding issues before the first alert sounds. These individuals are far different in skill and mindsets from the typical SOC occupant. They have an instinct to follow their noses, and a strong drive to find an explanation for every anomaly they identify. They usually possess the tenacity to not give up on an issue until they have an answer.

True, these individuals are more expensive, and much harder to find. When you do find them however and empower them to dig for issues rather than waiting on them to surface via an alert, they quickly pay for themselves.

Here are some tips for achieving a true investigative function:

Hire the right folks

This almost goes without saying, but having a team with investigative skills and mindset is critical to achieving an effective organization. I prioritize investigative abilities over actual experience in most cases, to find the people I need. Selection requires a careful interview process, because these special skills will often not jump out at the reader from a resume page.

Provide the needed tools

Although you don’t need to spend a fortune on tools for a good team, certain fundamental systems are essential for a good investigative function. The cornerstone for this is the Security Incident Event Management system (SIEM), which collects log records from various systems into a single repository. This requirement does not stop at the purchase of the system, however. It is critical that key servers, network devices, and even workstations be setup to send their log records to the SIEM. A SIEM with the necessary data allows an investigator to correlate events from various logs to look for behavioral patterns. Without an SIEM, the Analyst would need to look at too many individual logs, hampering the process.

Provide good training

Ongoing training is critical to a good investigative function. Sadly, I have seen few good training opportunities specific to this need. In my experience, training designed for red teams, which many companies use to test security from the outside of a company in, works best for investigators, given that it allows the Analysts to put themselves in the position of an attacker, so they have a better idea what to look for.

Give them time

A good investigator must have time to follow their gut. If they see something that bothers them, they must be allowed to dig into the issue until they find a problem or satisfy themselves that all is well. In the process, they will go down many rabbit holes and find nothing of note, but each one is an educational experience. Rather than scheduling them for too many routine activities, free them to do what they do best—dig.

Bottom line: An SOC is a useful part of our cybersecurity arsenal, but its main benefit will be in helping to minimize damage from an issue that has already happened. A strong investigative team, on the other hand, can help to identify and resolve issues before they cause major damage, which is always our preference.


Robert C. Covington, the "Go To Guy" for small and medium business security and compliance, is the founder and president of Mr. Covington has B.S. in Computer Science from the University of Miami, with over 30 years of experience in the technology sector, much of it at the senior management level. His functional experience includes major technology implementations, small and large-scale telecom implementation and support, and operations management, with emphasis on high-volume, mission critical environments. His expertise includes compliance, risk management, disaster recovery, information security and IT governance.

Mr. Covington began his Atlanta career with Digital Communications Associates (DCA), a large hardware/software manufacturer, in 1984. He worked at DCA for over 10 years, rising to the position of Director of MIS Operations. He managed the operation of a large 24x7 production data center, as well as the company’s product development data center and centralized test lab.

Mr. Covington also served as the Director of Information Technology for Innotrac, which was at the time one of the fastest growing companies in Atlanta, specializing in product fulfillment. Mr. Covington managed the IT function during a period when it grew from 5 employees to 55, and oversaw a complete replacement of the company’s systems, and the implementation of a world-class call center operation in less than 60 days.

Later, Mr. Covington was the Vice President of Information Systems for Teletrack, a national credit bureau, where he was responsible for information systems and operations, managing the replacement of the company’s complete software and database platform, and the addition of a redundant data center. Under Mr. Covington, the systems and related operations achieved SAS 70 Type II status, and received a high audit rating from the Federal Deposit Insurance Corporation and the Office of the Comptroller of the Currency.

Mr. Covington also served as Director of Information Technology at PowerPlan, a software company providing software for asset-intensive industries such as utilities and mining concerns, and integrating with ERP systems including SAP, Oracle Financials, and Lawson. During his tenure, he redesigned PowerPlan's IT infrastructure using a local/cloud hybrid model, implemented IT governance based on ITIT and COBIT, and managed the development of a new corporate headquarters.

Most recently, Mr. Covington, concerned about the growing risks facing small and medium business, and their lack of access to an experienced CIO, formed togoCIO, an organization focused on providing simple and affordable risk management and information security services.

Mr. Covington currently serves on the board of Act Together Ministries, a non-profit organization focused on helping disadvantaged children, and helping to strengthen families. He also leads technical ministries at ChristChurch Presbyterian. In his spare time, he enjoys hiking and biking.

The opinions expressed in this blog are those of Robert C. Covington and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.