Physical Security Risk and Countermeasures: Effectiveness Metrics

Is your security program working? Here's how to establish metrics for systematic measurement and improvement of countermeasures.

Also see "What Kind of Information Do We Need to Evaluate to Determine Security Program Effectiveness?" from the same chapter.

What Kind of Metrics Can Help Us Analyze Security Program Effectiveness?

There are several possible metrics to use. Each metric evaluates a different factor in security program effectiveness. These can be used in combination to achieve a complete picture of overall system effectiveness. Some metrics are useful for both new and existing security facilities, and others are only applicable to existing facilities.

  • Metrics usable for proposed security programs include:

    —Vulnerability/Countermeasure Matrix

    —Adversary Sequence Diagrams
  • Metrics usable for existing security programs include:

    —Adversary Sequence Diagrams

    —Vulnerability/Countermeasure Matrix

    —Security events logs

    —Patrol logs (vulnerabilities spotting/violations spotting)

    —Annual risk analysis

Each of these are explained below.

Adversary Sequence Diagrams

Adversary Sequence Diagrams relate to a specific type of threat actor—those who use intrusion to gain access to their target asset. The most valuable assets of organizations are not located at their front gate at street side. In order for an intruder to get to the target, the intruder must make his or her way from outside the property through various gates, doors, corridors, and then finally to the target. This is true whether the attacker is a terrorist, criminally violent threat actor, or economic or intellectual property criminal. It is true for all burglars, attackers using force or subversives. Whether the threat actor is breaking in, breaking down doors, or secretly making his or her way to an office during working hours to steal money or information, there is a common factor. Each attacker must make entry, make his or her way through passages and barriers, and arrive at the target. For most attackers, the plan is also to make their way back out again, without detection, if possible.

A sample adversary sequence diagram

Intrusion attackers come in three types:

1. Those using overwhelming force to make entry.

2. Those using stealth to make entry.

3. Those using the organization's normal business operations to make entry.

Obviously, each of these types presents different requirements for detection, assessment, and response. These three types also present themselves as two main types when encountering a response force:

  • Those who will surrender peacefully or try to flee (mostly economic criminals, petty criminals, and some violent criminals).
  • Those who will resist:

    —Those who will resist with moderate force (any threat actor except terrorists).

    —Those who will resist with overwhelming force (all terrorists and some violent criminals—only a few economic or petty criminals).

Intrusion threat actors can be further categorized into two broad groups:

1. Sophisticated criminals following an organized plan.

2. Opportunistic criminals mostly following their instincts (spontaneous planning).

Sophisticated criminals present special challenges for the following reasons:

  • Intrusions are generally well planned.
  • Sophisticated criminals know their target (its value, its location, the paths to the target, protective measures they will encounter on their way in and out).
  • Sophisticated criminals know your facility, including its daily operations.
  • They know your detection capabilities.
  • They know your security force quality, quantity, training, force capabilities, and weaknesses.
  • They can generally predict what your security response will be.
  • Except for terrorism, from an evidence standpoint, sophisticated criminals usually leave little evidence.

Unsophisticated criminals also present special challenges:

  • Unsophisticated criminals exhibit little or no preplanning, usually responding to opportunities without knowing much about their target, its detection capabilities, occupants, or its response capabilities.
  • Poor planning means they may not act predictably either in terms of what direction they go and in how they will respond when encountered by a response officer.
  • Unsophisticated criminals rarely make a prolonged entry for fear of detection and response.
  • From an evidence standpoint, unsophisticated criminals often leave a chaotic crime scene.

The key to dealing with intrusion threat actors is to detect them as early as possible and intercept them with a superior response before they can make their way to their intended target. Failing that, you can detect them and present a superior response on their exit.

This is where the design basis threat becomes relevant again. The quality of detection, assessment, and response should be proportionate to the level of threat actor and their worst-case scenario. Countermeasure selection must be appropriate to the sophistication and force of the design basis threat.

The Adversary Sequence Diagram (Figure 18.1) is used to evaluate the possible points of entry and the paths that a threat actor could take to his or her target, and then to the exit. This, of course, will result in multiple Adversary Sequence Diagrams, one for each entry/target combination.

The next type of metric is the Vulnerability/Countermeasure Matrix.

Vulnerability/Countermeasure Matrix

The Vulnerability/Countermeasure Matrix is a spreadsheet of vulnerabilities (rows) and various types of countermeasures (columns). Ideally, every vulnerability identified should be listed on its own row. The vulnerabilities can be categorized by major asset groups, buildings, areas, and so forth. At intersection points between vulnerabilities and countermeasures, place a percentage of probable mitigation (1 being 100%).

For example, for detection systems, if detection of an exploiter of this vulnerability is assured, that gets a 1. For assessment, if there is a camera that can verify the alarm, that gets a 1.

For response, if a response can be mounted that can intervene before the subject reaches his target, that gets a 1. This could, for example, be a security intercom that allows the console officer to intervene and interrogate the subject remotely, while a patrol officer is being sent. The subject's response will dictate further action. If the subject continues after being intercepted by an intercom call, that defines intent. Then three other options exist. These include denial, containment, and recovery.

For facilities where the acquisition of the asset could cause unacceptable consequences, such as chemical plants and nuclear power plants, denial is required. This requires a robust security force with excellent training.

For facilities where the mere acquisition of the asset is not a consequence, only its loss would be, containment is a possibility. This allows the security force time to plan a response, including staging a recovery on the aggressor's exit.

For sites where a robust response force is not financially feasible, recovery may be a legitimate option. For this, excellent evidence is required, including vehicle ID including license plate and a clear photo of intruders (face and clothes, height, weight, gender, etc.) and evidence of the crime and evidence of the removal and getaway. Few commercial security systems can accomplish this, though all should.

For evidence, if there is a camera that can get a good identification, that would receive a 1. For a camera that can read gender, clothing description, and other details but not facial identification, that might be a 0.5, and for a camera that can see form and movement but could not identify gender, that might be a 0.2.

It may be useful to assemble columns in the spreadsheet into groups. I have done this two ways but finally settled on the second. The first grouping included hi-tech, lo-tech, and no-tech countermeasures. The second grouping included access control, detection, assessment, delay, response, and evidence. I now use the second group because it explains the function of each countermeasure more clearly. Some countermeasures can serve multiple functions, which is not evident in the first categorization. For example, a video camera can detect and assess. Arguably it could even be considered a response if a pan/tilt camera is seen to move from one position to follow a subject. This would be observable by the subject and thus constitute a response that could be a deterrent.

[CSOonline editor's note: This image is rotated; otherwise the type would be too small to read.]

A sample countermeasures effectiveness matrix

You will notice that there is no category for deterrence, because deterrence cannot be easily measured or estimated. Deterrence is purely a subjective phenomenon completely reliant on the subject in question. Factors involved in deterrence include:

  • The subject's motivation.
  • The subject's determination.
  • The subject's concern for detection and capture (this is a key reason why terrorism is so difficult, because many terrorists do not care if they die in the attack, so they certainly expect to be detected and responded to). Thus, none of the usual factors comprise deterrence for a terrorist, except any response factors that could compromise the fulfillment of their mission.

Thus, if a subject wishes to elude detection, the deterrence value of alarms, dogs, cameras, lights, and so forth may be high. But if a subject has little concern for detection or response (terrorists, workplace violence threats, mental health threats, activist action groups, etc.), deterrent value of the same countermeasures may be low. One cannot estimate deterrence from the existence of countermeasures, and I do not recommend that you even consider it as a factor.

This principle should guide the design of response countermeasures. The degree of response should be directly correlated to the consequences of a threat action. If the consequences of a threat action are acceptable and can be mitigated after the loss, then the response can be unarmed and muted, such as in a normal office or commercial environment. However, if the consequences of the loss are wholly intolerable, such as at a nuclear power plant, nuclear weapons storage facility, or Phosgene (CG) chemical production plant, then the response capability should be superior to the severest possible threat action.

(continued)

For any very severe consequence that could affect community welfare (chemical plant, etc.), I recommend the Sandia Risk Assessment Model and very robust countermeasures. Sadly, this is most often not the case for facilities in the commercial sector which are not subject to strict government security regulations. In my opinion, the government should mandate stricter risk assessment and countermeasure programs at many more types of facilities than they do now, because many facilities with relatively relaxed risk assessment and countermeasure requirements constitute a grave risk to society.

By categorizing countermeasures by their functions, one can get a picture of the overall effectiveness of the countermeasures that relate to a specific vulnerability. Within the categories of entry control, you might include access control reader, vehicle checkpoint, and so forth. For detection, you might have DPS (door position switch) motion detector, motion video detector, fence line detector, buried perimeter detector, left-behind article detection, patrol detection, and dog. For assessment you might include video camera, intercom, and patrol officer. For delay you might include deployable barriers and burglar bars. For response you might include patrols, dispatch, and intercom. For evidence you might include video archiving, audio intercom archiving, telephone, and 911 logging recorder. For a given security program, there may be dozens of types of countermeasures.

Different countermeasures will be applicable to different types of vulnerabilities (e.g., a glass facade is vulnerable to blast and intrusion). Countermeasures could include glass break detectors, blast film CCTV cameras, crime prevention through environmental design (CPTED) measures providing blast standoff and so forth. Because certain vulnerabilities may apply to multiple threat actions, the range of possible countermeasures is not universally applicable. However, each countermeasure has an effectiveness factor against each threat. Glass break detectors are of no help to a blast threat, but they are helpful against burglary. It is appropriate to list all possible countermeasures and rate them each for effectiveness against the types of threats that they can mitigate. This provides an overall view of effectiveness. It may also be useful for high-consequence vulnerabilities to add a remarks field to the right of all columns to note the highest consequence, as this may be taken into consideration when preparing the qualitative report. Apply an effectiveness estimate to every applicable countermeasure. The value of layering countermeasures will begin to display itself as the value of each of the countermeasures begins to add to a value of 1.

Do this for every vulnerability listed in the vulnerability analysis until all applicable countermeasures for all vulnerabilities have been estimated. You will note that certain countermeasures are capable of mitigating multiple vulnerabilities, and most vulnerabilities require multiple countermeasures to fully mitigate. Also keep in mind the design basis threat. If the design basis threat is violent crime, countermeasures that will mitigate violence may be of little use to economic threats. For economic crime, countermeasures that could mitigate terrorism may be of little use. For complex projects, I have actually developed Vulnerability/Countermeasure Matrices for several types of threat actions. This is not usually done for every vulnerability in the facility, but for key assets (for violence, you may do this for people, but not for office equipment vulnerabilities).

The Vulnerability/Countermeasure Matrix should be prepared after the Adversary Sequence Diagrams, which will help to point out vulnerabilities that cannot be noticed without performing them.

Having explained this process to classes and individual consultants, it requires a step-by-step explanation to fully understand:

  • Step 1—Create a spreadsheet of every vulnerability in the project. This spreadsheet will also serve as the basis for the risk register, if the project requires it (see Chapter 10). For the purposes of illustration, we will look at 2. Each vulnerability will reside on its own row, separated by asset class or area of facility.
  • Step 2—Add columns for risk number, probability score, vulnerability score, consequences score, and risk score R = (P + V + C)/3. Add a column for recommended countermeasure and estimated cost.
  • Step 3—Create columns for each type of countermeasure, grouped by functions.

    Function classes include:

    —Entry control

    —Detection

    —Assessment

    —Delay

    —Response

    —Evidence

    To the left of each function, include a column marked Countermeasure Effectiveness Estimate (or CEE). To the right of all columns, add an additional column titled Total Mitigation Estimate (or TME) (Figure 18.2).
  • Step 4—For each vulnerability place an "X" under each countermeasure that applies. Then, for each function, place an estimate from 0 to 1 where 1 is total mitigation for that function and 0 is no mitigation for that function.

    For example, for detection, countermeasures might include door position switch, glass break detector, guard dog, and patrol. For each of these that are applicable for this vulnerability, place an X under the countermeasure. Then estimate the total mitigation for that vulnerability for the detection function. If detection is assured, place a 1 in the Estimate column. If detection is not likely to occur, place a 0 in the column. If detection is nearly always likely, place a number lower than 1 and more than 0.5 in the column. Do this for each vulnerability and for each functional group. This provides the mitigation score for each function for each vulnerability.

    Then place a weighting on the functional estimates. Since we have 6 functions, a balanced weighting would be 16.6% for each function.

    The formula for total mitigation is simple and straightforward. Total Mitigation = (Entry Control * Weighted Score {WS} Entry Control) + (Detection * WS Detection) + (Assessment * WS Assessment) + (Delay * WS Delay) + (Response * WS Response) + (Evidence * WS Evidence).

  • (continued)

    Security Event Logs

    Security event logs are also a very good way to determine overall security program effectiveness. Across a year's time, security logs will display trends and identify unresolved vulnerabilities. We are interested in both, but especially the unresolved vulnerabilities.

    It is unlikely that every last vulnerability will be identified in any risk assessment, but you can be certain that offenders will notice any unresolved vulnerabilities and try to exploit them. These will often be found by minor offenders or the guard staff. These minor exploits will show up as security events in the logs and are a valuable source for tightening up those unresolved vulnerabilities that would otherwise go unnoticed.

    I recommend that whatever logging method you use to keep track of security events should have a column to track whether each security event was related to an unresolved vulnerability. An additional column could identify the unresolved vulnerability. This allows the security director to relate security events either to misbehavior that was handled in accordance with policy or was an event that should spark reconsideration of security countermeasures.

    Over the course of a year, any unresolved vulnerabilities that develop into security events will draw management's attention to the needs for those vulnerabilities to be mitigated with appropriate countermeasures. Adding the column to identify the security event as related to an unresolved vulnerability and describing that vulnerability allows the security director to quickly identify any unresolved vulnerabilities and also note which vulnerabilities are related to recurring security events.

    The goal of ongoing risk assessments is to continuously uncover unresolved vulnerabilities and emerging threats and to make accommodations for them. Security event logs are one of the very best tools an analyst can use to achieve this goal.

    In the event that there is no column to identify if each security event is related to an unresolved vulnerability, all is not lost. An analyst can import the logs to a spreadsheet program and add the columns. If the analyst is familiar with the facility, he or she will likely think of the vulnerabilities that could relate to the security event. If not, he or she can assemble the related security events and then discuss these events with staff to uncover any unresolved vulnerabilities.

    The spreadsheet acts as a metric, listing both incidents related to vulnerabilities and those that are not. The percentage of incidents related to vulnerabilities is a useful metric to determine that the security program is minimizing risk.

    Patrol Logs (Vulnerabilities Spotting/Violations Spotting)

    In the same manner that security incident reports can uncover unresolved vulnerabilities, so too can patrol logs. Quality security program directors train their patrol officers to understand vulnerabilities and to spot them when they see them. I always find it interesting when performing risk analysis surveys that interviews with both post and patrol officers always and without exception uncover unresolved vulnerabilities.

    There is a wealth of information among the officers "on the ground" about the weaknesses in security countermeasures. It is very common after a major security incident to hear one or many officers say "Yeah, I knew that was going to happen someday." So why did they not report it? Usually it is because management does not emphasize focusing on vulnerabilities and reporting them to management.

    By training security officers to observe, and not just to see, management can find those vulnerabilities that are missed by risk analysts and management due to their lack of intimate familiarity with the facility and its operations. Security officers, who spend hours every day interacting with the business operations and every corner of the facility, know every vulnerability well. But in most cases, they are not trained to see them as vulnerabilities that should be addressed and reported to management.

    The patrol logs spreadsheet acts as a metric, listing both patrol notes that are related to vulnerabilities and those that are not. The percentage of patrol notes related to vulnerabilities is a useful metric to determine that the security program is minimizing risk.

    Annual Risk Analysis

    Finally, the risk analysis should be updated annually. This presents an opportunity once each year to compare overall risk progression year over year. The delta between this year and previous years serves as a useful metric to determine risk progression.

    Insider: How a good CSO confronts inevitable bad news
    Join the discussion
    Be the first to comment on this article. Our Commenting Policies