What Kind of Metrics Can Help Us Analyze Security Program Effectiveness?
There are several possible metrics to use. Each metric evaluates a different factor in security program effectiveness. These can be used in combination to achieve a complete picture of overall system effectiveness. Some metrics are useful for both new and existing security facilities, and others are only applicable to existing facilities.
Metrics usable for proposed security programs include:
—Vulnerability/Countermeasure Matrix—Adversary Sequence Diagrams
Metrics usable for existing security programs include:
—Adversary Sequence Diagrams
—Security events logs
—Patrol logs (vulnerabilities spotting/violations spotting)—Annual risk analysis
Each of these are explained below.
Adversary Sequence Diagrams
Adversary Sequence Diagrams relate to a specific type of threat actor—those who use intrusion to gain access to their target asset. The most valuable assets of organizations are not located at their front gate at street side. In order for an intruder to get to the target, the intruder must make his or her way from outside the property through various gates, doors, corridors, and then finally to the target. This is true whether the attacker is a terrorist, criminally violent threat actor, or economic or intellectual property criminal. It is true for all burglars, attackers using force or subversives. Whether the threat actor is breaking in, breaking down doors, or secretly making his or her way to an office during working hours to steal money or information, there is a common factor. Each attacker must make entry, make his or her way through passages and barriers, and arrive at the target. For most attackers, the plan is also to make their way back out again, without detection, if possible.
Intrusion attackers come in three types:
1. Those using overwhelming force to make entry.
2. Those using stealth to make entry.
3. Those using the organization's normal business operations to make entry.
Obviously, each of these types presents different requirements for detection, assessment, and response. These three types also present themselves as two main types when encountering a response force:
- Those who will surrender peacefully or try to flee (mostly economic criminals, petty criminals, and some violent criminals).
Those who will resist:
—Those who will resist with moderate force (any threat actor except terrorists).—Those who will resist with overwhelming force (all terrorists and some violent criminals—only a few economic or petty criminals).
Intrusion threat actors can be further categorized into two broad groups:
1. Sophisticated criminals following an organized plan.
2. Opportunistic criminals mostly following their instincts (spontaneous planning).
Sophisticated criminals present special challenges for the following reasons:
- Intrusions are generally well planned.
- Sophisticated criminals know their target (its value, its location, the paths to the target, protective measures they will encounter on their way in and out).
- Sophisticated criminals know your facility, including its daily operations.
- They know your detection capabilities.
- They know your security force quality, quantity, training, force capabilities, and weaknesses.
- They can generally predict what your security response will be.
- Except for terrorism, from an evidence standpoint, sophisticated criminals usually leave little evidence.
Unsophisticated criminals also present special challenges:
- Unsophisticated criminals exhibit little or no preplanning, usually responding to opportunities without knowing much about their target, its detection capabilities, occupants, or its response capabilities.
- Poor planning means they may not act predictably either in terms of what direction they go and in how they will respond when encountered by a response officer.
- Unsophisticated criminals rarely make a prolonged entry for fear of detection and response.
- From an evidence standpoint, unsophisticated criminals often leave a chaotic crime scene.
The key to dealing with intrusion threat actors is to detect them as early as possible and intercept them with a superior response before they can make their way to their intended target. Failing that, you can detect them and present a superior response on their exit.
This is where the design basis threat becomes relevant again. The quality of detection, assessment, and response should be proportionate to the level of threat actor and their worst-case scenario. Countermeasure selection must be appropriate to the sophistication and force of the design basis threat.
The Adversary Sequence Diagram (Figure 18.1) is used to evaluate the possible points of entry and the paths that a threat actor could take to his or her target, and then to the exit. This, of course, will result in multiple Adversary Sequence Diagrams, one for each entry/target combination.
The next type of metric is the Vulnerability/Countermeasure Matrix.
The Vulnerability/Countermeasure Matrix is a spreadsheet of vulnerabilities (rows) and various types of countermeasures (columns). Ideally, every vulnerability identified should be listed on its own row. The vulnerabilities can be categorized by major asset groups, buildings, areas, and so forth. At intersection points between vulnerabilities and countermeasures, place a percentage of probable mitigation (1 being 100%).
For example, for detection systems, if detection of an exploiter of this vulnerability is assured, that gets a 1. For assessment, if there is a camera that can verify the alarm, that gets a 1.
For response, if a response can be mounted that can intervene before the subject reaches his target, that gets a 1. This could, for example, be a security intercom that allows the console officer to intervene and interrogate the subject remotely, while a patrol officer is being sent. The subject's response will dictate further action. If the subject continues after being intercepted by an intercom call, that defines intent. Then three other options exist. These include denial, containment, and recovery.
For facilities where the acquisition of the asset could cause unacceptable consequences, such as chemical plants and nuclear power plants, denial is required. This requires a robust security force with excellent training.
For facilities where the mere acquisition of the asset is not a consequence, only its loss would be, containment is a possibility. This allows the security force time to plan a response, including staging a recovery on the aggressor's exit.
For sites where a robust response force is not financially feasible, recovery may be a legitimate option. For this, excellent evidence is required, including vehicle ID including license plate and a clear photo of intruders (face and clothes, height, weight, gender, etc.) and evidence of the crime and evidence of the removal and getaway. Few commercial security systems can accomplish this, though all should.
For evidence, if there is a camera that can get a good identification, that would receive a 1. For a camera that can read gender, clothing description, and other details but not facial identification, that might be a 0.5, and for a camera that can see form and movement but could not identify gender, that might be a 0.2.
It may be useful to assemble columns in the spreadsheet into groups. I have done this two ways but finally settled on the second. The first grouping included hi-tech, lo-tech, and no-tech countermeasures. The second grouping included access control, detection, assessment, delay, response, and evidence. I now use the second group because it explains the function of each countermeasure more clearly. Some countermeasures can serve multiple functions, which is not evident in the first categorization. For example, a video camera can detect and assess. Arguably it could even be considered a response if a pan/tilt camera is seen to move from one position to follow a subject. This would be observable by the subject and thus constitute a response that could be a deterrent.
[CSOonline editor's note: This image is rotated; otherwise the type would be too small to read.]
You will notice that there is no category for deterrence, because deterrence cannot be easily measured or estimated. Deterrence is purely a subjective phenomenon completely reliant on the subject in question. Factors involved in deterrence include:
- The subject's motivation.
- The subject's determination.
- The subject's concern for detection and capture (this is a key reason why terrorism is so difficult, because many terrorists do not care if they die in the attack, so they certainly expect to be detected and responded to). Thus, none of the usual factors comprise deterrence for a terrorist, except any response factors that could compromise the fulfillment of their mission.
Thus, if a subject wishes to elude detection, the deterrence value of alarms, dogs, cameras, lights, and so forth may be high. But if a subject has little concern for detection or response (terrorists, workplace violence threats, mental health threats, activist action groups, etc.), deterrent value of the same countermeasures may be low. One cannot estimate deterrence from the existence of countermeasures, and I do not recommend that you even consider it as a factor.
This principle should guide the design of response countermeasures. The degree of response should be directly correlated to the consequences of a threat action. If the consequences of a threat action are acceptable and can be mitigated after the loss, then the response can be unarmed and muted, such as in a normal office or commercial environment. However, if the consequences of the loss are wholly intolerable, such as at a nuclear power plant, nuclear weapons storage facility, or Phosgene (CG) chemical production plant, then the response capability should be superior to the severest possible threat action.
For any very severe consequence that could affect community welfare (chemical plant, etc.), I recommend the Sandia Risk Assessment Model and very robust countermeasures. Sadly, this is most often not the case for facilities in the commercial sector which are not subject to strict government security regulations. In my opinion, the government should mandate stricter risk assessment and countermeasure programs at many more types of facilities than they do now, because many facilities with relatively relaxed risk assessment and countermeasure requirements constitute a grave risk to society.
By categorizing countermeasures by their functions, one can get a picture of the overall effectiveness of the countermeasures that relate to a specific vulnerability. Within the categories of entry control, you might include access control reader, vehicle checkpoint, and so forth. For detection, you might have DPS (door position switch) motion detector, motion video detector, fence line detector, buried perimeter detector, left-behind article detection, patrol detection, and dog. For assessment you might include video camera, intercom, and patrol officer. For delay you might include deployable barriers and burglar bars. For response you might include patrols, dispatch, and intercom. For evidence you might include video archiving, audio intercom archiving, telephone, and 911 logging recorder. For a given security program, there may be dozens of types of countermeasures.
Different countermeasures will be applicable to different types of vulnerabilities (e.g., a glass facade is vulnerable to blast and intrusion). Countermeasures could include glass break detectors, blast film CCTV cameras, crime prevention through environmental design (CPTED) measures providing blast standoff and so forth. Because certain vulnerabilities may apply to multiple threat actions, the range of possible countermeasures is not universally applicable. However, each countermeasure has an effectiveness factor against each threat. Glass break detectors are of no help to a blast threat, but they are helpful against burglary. It is appropriate to list all possible countermeasures and rate them each for effectiveness against the types of threats that they can mitigate. This provides an overall view of effectiveness. It may also be useful for high-consequence vulnerabilities to add a remarks field to the right of all columns to note the highest consequence, as this may be taken into consideration when preparing the qualitative report. Apply an effectiveness estimate to every applicable countermeasure. The value of layering countermeasures will begin to display itself as the value of each of the countermeasures begins to add to a value of 1.