Before we can address the question of how you can tell if your vulnerability management program is failing, we must answer a basic question: What is a vulnerability management program?
A vulnerability management program is a program that identifies vulnerabilities in an organization's network; monitors and tracks the remediation of these vulnerabilities; analyzes the root cause(s) of these vulnerabilities; and makes strategic changes to current processes in order to fix the root cause(s) of the vulnerabilities. For example, vulnerabilities are identified by performing some sort of assessment like a vulnerability assessment. The vulnerabilities are reviewed and the system owner of a vulnerable device is contacted in order to notify them that their system has a vulnerability which must be remediated. The system owner remediates the vulnerability and lets someone in the vulnerability management role know the vulnerability has been remediated.
Next the root cause of the vulnerability is reviewed in order to determine why the vulnerability was present in the network (i.e., a missing patch, web admin console with default password, SSLv2, etc.). The organization's current process (patch management process, minimum security baseline documentation, policies and procedures, etc.) is modified in order to prevent the vulnerability from reappearing in the future.
Most vulnerability management programs heavily rely on network and web application scanners, although these technologies should be supplemented by manual assessments such as attack and penetration testing and grey box web application assessments. A mature vulnerability management program is critical for any security program to be successful. If implemented correctly, a vulnerability management program can help to identify weaknesses in an organization's patch management program, firewall and router configurations, minimum security baselines, policies and procedures, web application developer training, etc. On the other hand, a poorly developed vulnerability management program can lead to an overall false sense of security.
The top indications an organization's vulnerability management program is failing are as follows:
The Same Vulnerabilities Appear on the Same Host Every Month
As a security consultant, I have performed my share of quarterly vulnerability assessments for many of our clients. Many times I find a critical vulnerability on one of the client's systems, and contact them to inform them that I recommend the vulnerability be addressed as soon as possible. Three months pass, and once again it is time to perform a vulnerability assessment on the client I found the critical vulnerability on three months earlier. To my amazement, I find that the critical vulnerability I had said should be addressed as soon as possible had not been addressed.
The idea behind a vulnerability management program is that once a vulnerability is discovered, it is remediated in a reasonable amount of time. Any successful vulnerability management program must have policies in place which require system owners address vulnerabilities that are found during a security assessment within a reasonable amount of time. These policies should include timeframes for how long a system owner has to address a vulnerability, and should be directly tied to the severity of the particular vulnerability. A critical vulnerability that could enable an attacker to run arbitrary commands on the underlying operating system of the affected system should be addressed within a very short timeframe, while a less critical vulnerability does not need to be remediated as quickly. In any case, with very few exceptions, if a high risk vulnerability still is present in your network three months after it was identified, your vulnerability management Program is not effective. M/p>
How to address this problem:
Create policies which force system owners to address vulnerabilities that are found on their systems. These policies should be directly tied to the severity of the vulnerability.
The same categories of vulnerabilities appear every month.
Client: There has been a mistake& we fixed the SSLv2 vulnerability last quarter.
Consultant: That was a different machine. The vulnerability scanner found this vulnerability on machine 10.0.0.1.
Client: That makes sense. That is a new machine that we placed on the network a month ago; I will have someone update the httpd.conf file to disable SSLv2.
Any successful vulnerability management Program has both reactive and proactive components to it. A reactive component without a proactive component will result in many headaches, fire drills, and long nights. A proactive component without a reactive component results in a network filled with vulnerabilities which have never been addressed. A quick breakdown of these two components is as follows:
Reactive: A reactive component means that vulnerabilities that are identified must be remediated within reasonable timeframes. As the name suggests, the company is reacting to the results of a security assessment by remediating the vulnerabilities that are identified.
Proactive: A proactive component means that when a vulnerability is identified, the root cause of the vulnerability is identified and addressed. In the case of vulnerabilities that are the result of poor system hardening (SSLv2, Dangerous HTTP Methods, Default Error Handling, etc.), the minimum security baseline (MSB) documentation should be updated and / or policies need to be enforced which require current MSBs be applied to any system before it is placed on the network. If critical patches are missing, the current patch management program should be reviewed in order to identify why it is failing to patch specific systems and / or services. The idea behind a proactive component in a vulnerability management program is to isolate and fix the root cause of a problem. A proactive component is needed so the same categories of vulnerabilities will not appear every month.
How to address this problem:
Root cause analysis must be performed on vulnerabilities that are found in the course of a security assessment. Once the root cause of the problem is found, then processes should be adjusted in order to make sure the vulnerability does not appear in the network again. Process adjustments include changes to the patch management program, changes to the current minimum security baselines, updates to group policy, etc.
The person responsible for the vulnerability management program is on Valium
A well designed Vulnerability Management Program should function like a well oiled machine. Before a Vulnerability Assessment is even performed, systems in the environment should have system owners assigned to them; the system owner for each system should be easy to locate; and policies should be developed which force system owners to address vulnerabilities on systems for which they are responsible. Root cause analysis should be used to identify why specific vulnerabilities are present; and over time changes should be made to processes to help to eliminate vulnerabilities before they surface in the production environment.
Over time as these things are fully implemented, the job of the person responsible for the Vulnerability Management Program should become less chaotic. If you find yourself trying to track down a system owner; fighting with a system owner to fix a specific vulnerability; or seeing the same vulnerabilities present in your network month after month, then you know that your Vulnerability Management Program is not as effective as it could be.
How to address this problem:
Before you perform a security assessment it is important that you do the following: know who owns specific devices in your network; have policies in place that force system owners to remediate vulnerabilities; put processes in place which require root cause analysis be performed on identified vulnerabilities; and update processes in order to make sure a vulnerability does not appear again.
You fail your PCI-approved scanning vendor (ASV) scan two weeks after performing a vulnerability assessment
I have performed an external vulnerability assessment on the external presence of some of our clients in preparation for their PCI ASV scans. This assessment did not identify any vulnerabilities which would cause the client to fail their upcoming PCI ASV scan. Two weeks later, the ASV scan is run against the clients externally facing PCI zone. The first attempt at the ASV scan identified vulnerabilities which resulted in a failing PCI ASV scan. Upon further investigation, the client found that in the two weeks between the ASV scan and the preparatory vulnerability assessment, another server was placed in the production environment which had not been properly hardened. Vulnerabilities were identified on this server which caused the client to fail their PCI ASV Scan.
This scenario is a major problem with Vulnerability Management Programs that do not take a proactive approach to security. If solid Minimum Security Baselines were in place and policies which require the application of these MSBs before a system is placed in a production environment also were in place, then a new server would not result in additional vulnerabilities being present in the network. The ultimate goal of security is to help to reduce risk. If your Vulnerability Management Program only identifies vulnerabilities once they are in a production environment, then it will be extremely limited in its overall effectiveness.
How to address this problem:
When vulnerabilities are identified, processes need to be updated to fix not only the vulnerabilities, but also the broken process which resulted in the vulnerabilities. Broken processes could include the patch management process, system hardening guidelines, etc.
You have no solid numbers showing your program is working
It is important to track your vulnerability management program's metrics. Metrics like the number, severity, and category of vulnerabilities that are identified during your Assessments should be tracked. It also is important to track the total vulnerabilities that have been remediated and the length of time it took to address these vulnerabilities.
Without numbers, you will not be able to perform effective trending analysis; and you will be unable to measure the effectiveness of your vulnerability management program. To clarify my point, imagine having the following conversation with your boss:
CIO: How effective is our vulnerability management program?
You: I think it's pretty effective.
CIO: Are we more secure than we were this time last year?
You: I am not sure, but I just remediated five vulnerabilities this week.
The result of the above conversation may not be so favorable. Further questions like, "Why should I continue our vulnerability management program if we are unable to verify it is working?" or "Why should I approve this budget request if you do not know if this new scanner will help to improve our overall threat posture?" may soon follow.
How to address this problem:
Metrics relating to the organization's vulnerability management program should be tracked. Metrics such as these should be tracked: the number of vulnerabilities that are identified during security assessments; the severity of these vulnerabilities; the time it took for a vulnerability to be remediated; the category of identified vulnerabilities; the root cause of the vulnerability; the device the vulnerability was found on, and the number of false positives.
The vulnerability scanner gives inconsistent results
You wake up in the morning, take a shower, get dressed, jump in your car, jump on the highway, drive fifteen miles, exit at mile marker 30, make a left and you are at work. Suppose the next morning you did the EXACT same thing: took the exact same highway to the exact same exit, and found yourself at McDonalds? Needless to say, you probably would be a little confused. As humans we expect that if we perform the exact same steps we performed last time, and nothing has changed in our environment, then we should have the same outcome.
This example may sound a bit silly, but in all reality a poor vulnerability management program can be just as frustrating. I have encountered these types of frustrations while performing a vulnerability assessment against devices on a poorly-developed network architecture, or on web application servers which are prone to resource starvation. When performing these assessments, one scan may show that 15 servers are online; a minute later the scanner may show that 17 servers are online; and a third scan shows that only 10 devices are online. In these cases, it is vital to adjust your vulnerability scanners options in order to accommodate these scenarios. Dropping the total number of threads a vulnerability scanner uses from 100 to 10 sometimes is necessary in order to get consistent results.
Another scenario that can lead to inconsistent results is changing the scanners you use to perform your vulnerability scans each month. Your results will be inconsistent if one month you use Acunetix for web application scanning, the next month you use WebInspect, and another month you use AppScan. It is important to be consistent with the tools used to perform Vulnerability Scanning.
How to address this problem: