In part 3 of our metrics series, we discussed we how KRIs help identify risks while KPIs help us measure them. In this, our final article in the series, we’ll build on this knowledge to create metrics based on our four-stage model for qualifying risks and threats that we introduced in part 2. By making your way through our sequential process, you’ll be able to extract performance indicators that can enlighten your executives on security risks and management.
Chris Hodson
Figure 1. Four-stage model for qualifying risks and threats.
Keep in mind that the metrics that come from the four-stage process should be kept within the infosec team, and used to inform future KRIs. Why? Because the results that are generated by these steps are often in jargon and tech-speak that the board won’t understand – like this:
“We’ve seen a persistent 200 gbps DDoS attack targeting the external interface of our DMZ.”
“Our sandbox devices have been evaded five times through malware with a sophisticated form of payload obfuscation.”
These are not risks: they are threats. If a threat cannot exploit a vulnerability, who cares? If the vulnerability doesn’t impact the confidentiality, integrity or availability of data, so what?
For example, if we’re performing our Business Impact Assessments (BIAs) and we establish a maximum tolerable downtime, it’s the KPIs from our systems that keep us honest. If we have a KRI associated with “shadow IT consumption and data loss”, it’s our KPIs around the percentage of unapproved applications deployed across the estate that provide proof points. In summary:
- Business strategy tells you the direction the company is going in and how it will get there
- BIAs uncover the important assets necessary to support the strategy
- KRIs identify areas of risk which could prohibit a company achieving objectives
- KPIs measure and keep you honest
Sound confusing? I hope not – check out Figure 2 where I represented where and how business goals are mapped into risk and performance indicators:
Chris Hodson
Figure 2. Business Initiatives translate into Indicators.
Step 1: Qualify assets
We cannot apply the same level of security to everything (to do so would be very expensive and may increase friction in the user experience, which is unnecessary). We will apply stronger controls to systems and apps that contain stuff/data that we care about it.
To assess if we care about stuff/data, we leverage the BIA and data classification framework we discussed in above. This is where business continuity and security can align. In your BIAs, define location and technology stack inventory information for all systems that house confidential data, employee records, or systems where the confidentiality, integrity and/or availability requirements are “Critical/High.” Prioritize these first. The outputs of your BIA will give you pertinent information regarding the volume and sensitivity of data assets in your estate. The methodology varies but outputs generally include:
- Knowing how long you can afford to be without the data in question: Recovery Time Objective (RTO)
- Knowing how much data you can afford to lose in the event of a compromise: Recovery Point Objective
- Classification Requirements for Information: Confidentiality, Integrity and Availability.
When you are qualifying these assets, visibility is of paramount importance. We cannot protect what we cannot see. Challenges with visibility stem from four contemporary challenges:
- The rise of TLS traffic
- Multiple device types connecting to our network
- Cloud application consumption
- Increased regulatory and legal pressure
Let’s say you find 275 laptops in your environment that you don’t know anything about. There may be a legitimate business justification for these unapproved devices. If so, then you can build an argument for investing in a suitable control (which we discuss in step 4 below).
All too often, we exclusively think of an ‘asset’ as a physical device (laptops, phones); data are an asset too; Since we’re soon going to be forced to notify a supervisory authority within 72 hours of a known breach, a solid metric here would be a blue team exercise associated with some incident response scenarios which would provide stakeholder confidence that expedient report can be achieved:
- How long did it take to penetrate the environment?
- How long to discover data being ex-filtrated?
- How long before we had a public statement and briefed relevant board members?
Infosec management has to be a holistic exercise, an imperative consideration being to secure your supply chain: Are you comfortable knowing who is connecting to your network? Let’s build a picture of who is connecting, using metrics associated with vendors who have passed a third-party due diligence process, and who have ISO 27001 certification.
- How many external repositories house personal data?
- How many suppliers have completed your vendor and supplier readiness questionnaire?
- % of suppliers who are PCI-DSS and ISO-27001 certified.
Reports suggest that 75 percent of the web will be encrypted by 2019. Given this, if your organization is still waiting to deploy TLS interception, perhaps we should look to get the visibility heat map flashing red! The total volume of web-traffic encrypted-v-unencrypted through Web and Email Gateways is always a good starting point to justify the need for TLS interception. If you need some assistance with your business case, check out some produced that shows the rise of threats in SSL.
Step 2: Profile threats
At this point, we know what we are protecting and we know who is connecting and from which devices. Next, we need to better understand who is attacking us. In an ideal world, the security professional would provide a percentage likelihood of a data breach, but in reality, cyber security presents a volume of variables that cannot be measured as absolutes.
For example:
- Attribution of threat actor location is onerous due to anonymizing services and botnets.
- Security control strength and application consistency varies across organizations.
- Forensic breach analysis information is not publicly provided.
That said, we can generally break down threat actors into two categories. We then develop metrics associated with the ‘tools, techniques and procedures’ (TTPs) that actor actor exhibits:
Accidental
Insiders, third parties, admins who make a mistake (who could forget the Amazon S3 debacle?) A solid metric here is your recovery point objective and recovery time objective – how long can you afford for a system to be offline? Which threat events could cause such an outage?
Adversarial
Who is attacking us? How well funded are they? Are they committed and capable? Are you able to review previous incident reports and identify the tools, techniques and procedures used? Are others in your industry being targeted by this actor? Are indicators of compromise (IPS signatures, file hashes, PCAPs) available which help identify malicious activity?
There is a lot of talk at the moment regarding threat intelligence, and understandably so. Is your organization pulling in threat intelligence information from open source or commercial providers? Are you participating in industry working groups? It is my view that no single organization can achieve a satisfactory understand of the threat landscape, companies must collaborate. It might feel like a truism but with the attackers only needing to be right once and the customer needing to be right all the time, we need all the help we can get. In any walk of life, collaboration and co-operation is almost impossible if we are speaking different languages; this has long been an issue with cyber information sharing. This series isn’t intended to deep-dive into cyber threat intelligence but I recommend that any organization interested in obtaining the most thorough picture of the cyber threat landscape as possible, looks to obtain feeds in a normalized fashion. The Structured Threat Information Expression (STIX) provides a standardized, machine-readable language for threat information exchange and follows a number of the language components I cover in this series (Actors, Events, Vulnerabilities, etc).
Step 3: Assess vulnerabilities
From my experience, assessing vulnerabilities is our biggest challenge. As security professionals, we talk about “people, process, and technology” often. Unfortunately, we focus too heavily on technology, nowhere more so than with vulnerability management. If we take an automated vulnerability scanner and point it at a network, invariably we’ll see metrics of a sort. But much is the response is a cacophony of noise – lot of warnings, with no way to prioritize them.
A great method for creating metrics on vulnerabilities comes from the tried-and-tested Common Vulnerability Scoring System (CVSS).[1] You can see a visualization of that here on the FIRST website that breaks metrics down into three groups: base, temporal, end environmental metrics.
Under this rubric, the “Base Metric Group” represents the characteristics of a vulnerability, including exploitability and impact metrics. In the “Temporal Metric Group,” you’ll find characteristics of vulnerabilities that can change over time but do not cross over to the user environments. For instance, a simple-to-use exploit kit was present, this would increase the CVSS score; on the other hand, the creation of an official patch would decrease the score. The “Environmental Metric Group” includes vulnerability characteristics that are unique to your environment; this is where the infosec team earn their salary: through understanding the vulnerability and the methods by which it (vulnerability) can be exploited, we can assess if controls currently exist which mitigate or remove the threat. This process assists in removing the ‘alert fatigue’ associated with 254,000 systems need patching!
These metrics allow security team analysts to customize CVSS scores based on the importance of affected IT assets to the organization. The metrics are measured based on complementary or alternative security controls in place.
While it’s not quite the same as responding to the question, “Are we secure? ”, the scores are a method of saying that we have controls for threats. We might use CVSS scores within the security team, but we can also use them with a broader audience to assess the efficacy of a control and produce a temporal score – and in this way, provide qualitative feedback. I have worked with organisations who take CVSS scores (quantitative) and apply a traffic light system (qualitative) when reporting an aggregated view of their “threat landscape.”
Metrics for technology solutions are one thing. But how do we measure vulnerabilities in people and process? We could use end-user awareness training used to produce metrics, but do the measurements still apply? We used to use a percentage of users who clicked on links? In a world of drive-by-downloads, malvertising and rouge Domain Validated Certs, is it any longer fair to castigate users for clicking on dodgy links or being lured to legitimate looking websites? All is not lost, we can apply some metrics to the people and process side of infosec (although generally with the support of ‘technology’). Check out this post I wrote on the subject.
We spoke earlier in the article around the time taken to identify a data breach or malicious insider, that’s a pretty solid metric for infosec efficacy. Another one I’ve used in the past is number of high/medium vulnerability findings across a penetration test or application code review. Take this information and work with your development teams; help them understand what, say, Cross Site Request Forgery actually is and how to remove it from your code.
You cannot measure what you don’t manage, right? Establish baselines for each of your operational activities and ensure that these are adhered to. I have always been a fan of the ISO 27001 Information Security Management System (ISMS). If you’re applying a governance model of policy, standard, guideline then why not leverage your automated vulnerability scanner and report on technical non-compliance to your guidelines?
Let’s talk about risk management processes: how many exceptions are you having to raise? Why? Does the policy or standard need to change? Given that our policies are driven from a series of business requirements, reporting an aggregated view of how well your organization complies with policy is a fantastic metric! Remember, security is iterative. If a policy isn’t working, or the implementation of a control is overly onerous, reevaluate!
Step 4. Apply controls: coverage is key, visibility is essential
When considering controls, ask yourself, “Is it worth it?” from a cost perspective or from an efficacy-of-solution perspective.
Controls are there to lessen the impact or likelihood of a risk occurring. Controls do not have to be absolute. It’s unusual for a control to remove a risk entirely. We’re looking to lessen the risk to a palatable level. Who sets this bar? It’s the business!
Organizations may decide that they prefer a less-effective control that offers 100% coverage of users and data, rather than a more effective system that doesn’t cover them well at all. Questions (which can be applied as metrics) I like to ask myself when considering a control:
- Will it cover all locations: Head office, branch locations, mobile workers?
- Can all my users benefit?
- Does the control have technical prerequisites: OS version, hardware requirements?