Nine (and a half) signs your vulnerability management program is failing

Before we can address the question of how you can tell if your vulnerability management program is failing, we must answer a basic question: What is a vulnerability management program?

A vulnerability management program is a program that identifies vulnerabilities in an organization’s network; monitors and tracks the remediation of these vulnerabilities; analyzes the root cause(s) of these vulnerabilities; and makes strategic changes to current processes in order to fix the root cause(s) of the vulnerabilities. For example, vulnerabilities are identified by performing some sort of assessment like a vulnerability assessment. The vulnerabilities are reviewed and the system owner of a vulnerable device is contacted in order to notify them that their system has a vulnerability which must be remediated. The system owner remediates the vulnerability and lets someone in the vulnerability management role know the vulnerability has been remediated.

Next the root cause of the vulnerability is reviewed in order to determine why the vulnerability was present in the network (i.e., a missing patch, web admin console with default password, SSLv2, etc.). The organization’s current process (patch management process, minimum security baseline documentation, policies and procedures, etc.) is modified in order to prevent the vulnerability from reappearing in the future.

Most vulnerability management programs heavily rely on network and web application scanners, although these technologies should be supplemented by manual assessments such as attack and penetration testing and grey box web application assessments. A mature vulnerability management program is critical for any security program to be successful. If implemented correctly, a vulnerability management program can help to identify weaknesses in an organization’s patch management program, firewall and router configurations, minimum security baselines, policies and procedures, web application developer training, etc. On the other hand, a poorly developed vulnerability management program can lead to an overall false sense of security.

The top indications an organization’s vulnerability management program is failing are as follows:

The Same Vulnerabilities Appear on the Same Host Every Month

As a security consultant, I have performed my share of quarterly vulnerability assessments for many of our clients. Many times I find a critical vulnerability on one of the client’s systems, and contact them to inform them that I recommend the vulnerability be addressed as soon as possible. Three months pass, and once again it is time to perform a vulnerability assessment on the client I found the critical vulnerability on three months earlier. To my amazement, I find that the critical vulnerability I had said should be addressed as soon as possible had not been addressed.

The idea behind a vulnerability management program is that once a vulnerability is discovered, it is remediated in a reasonable amount of time. Any successful vulnerability management program must have policies in place which require system owners address vulnerabilities that are found during a security assessment within a reasonable amount of time. These policies should include timeframes for how long a system owner has to address a vulnerability, and should be directly tied to the severity of the particular vulnerability. A critical vulnerability that could enable an attacker to run arbitrary commands on the underlying operating system of the affected system should be addressed within a very short timeframe, while a less critical vulnerability does not need to be remediated as quickly. In any case, with very few exceptions, if a high risk vulnerability still is present in your network three months after it was identified, your vulnerability management Program is not effective. M/p>

How to address this problem:

Create policies which force system owners to address vulnerabilities that are found on their systems. These policies should be directly tied to the severity of the vulnerability.

The same categories of vulnerabilities appear every month. Client: There has been a mistake& we fixed the SSLv2 vulnerability last quarter. Consultant: That was a different machine. The vulnerability scanner found this vulnerability on machine 10.0.0.1. Client: That makes sense. That is a new machine that we placed on the network a month ago; I will have someone update the httpd.conf file to disable SSLv2.

Any successful vulnerability management Program has both reactive and proactive components to it. A reactive component without a proactive component will result in many headaches, fire drills, and long nights. A proactive component without a reactive component results in a network filled with vulnerabilities which have never been addressed. A quick breakdown of these two components is as follows:

Reactive: A reactive component means that vulnerabilities that are identified must be remediated within reasonable timeframes. As the name suggests, the company is reacting to the results of a security assessment by remediating the vulnerabilities that are identified.

Proactive: A proactive component means that when a vulnerability is identified, the root cause of the vulnerability is identified and addressed. In the case of vulnerabilities that are the result of poor system hardening (SSLv2, Dangerous HTTP Methods, Default Error Handling, etc.), the minimum security baseline (MSB) documentation should be updated and / or policies need to be enforced which require current MSBs be applied to any system before it is placed on the network. If critical patches are missing, the current patch management program should be reviewed in order to identify why it is failing to patch specific systems and / or services. The idea behind a proactive component in a vulnerability management program is to isolate and fix the root cause of a problem. A proactive component is needed so the same categories of vulnerabilities will not appear every month.

How to address this problem:

Root cause analysis must be performed on vulnerabilities that are found in the course of a security assessment. Once the root cause of the problem is found, then processes should be adjusted in order to make sure the vulnerability does not appear in the network again. Process adjustments include changes to the patch management program, changes to the current minimum security baselines, updates to group policy, etc.

The person responsible for the vulnerability management program is on Valium

A well designed Vulnerability Management Program should function like a well oiled machine. Before a Vulnerability Assessment is even performed, systems in the environment should have system owners assigned to them; the system owner for each system should be easy to locate; and policies should be developed which force system owners to address vulnerabilities on systems for which they are responsible. Root cause analysis should be used to identify why specific vulnerabilities are present; and over time changes should be made to processes to help to eliminate vulnerabilities before they surface in the production environment.

Over time as these things are fully implemented, the job of the person responsible for the Vulnerability Management Program should become less chaotic. If you find yourself trying to track down a system owner; fighting with a system owner to fix a specific vulnerability; or seeing the same vulnerabilities present in your network month after month, then you know that your Vulnerability Management Program is not as effective as it could be.

How to address this problem:

Before you perform a security assessment it is important that you do the following: know who owns specific devices in your network; have policies in place that force system owners to remediate vulnerabilities; put processes in place which require root cause analysis be performed on identified vulnerabilities; and update processes in order to make sure a vulnerability does not appear again.

You fail your PCI-approved scanning vendor (ASV) scan two weeks after performing a vulnerability assessment

I have performed an external vulnerability assessment on the external presence of some of our clients in preparation for their PCI ASV scans. This assessment did not identify any vulnerabilities which would cause the client to fail their upcoming PCI ASV scan. Two weeks later, the ASV scan is run against the clients externally facing PCI zone. The first attempt at the ASV scan identified vulnerabilities which resulted in a failing PCI ASV scan. Upon further investigation, the client found that in the two weeks between the ASV scan and the preparatory vulnerability assessment, another server was placed in the production environment which had not been properly hardened. Vulnerabilities were identified on this server which caused the client to fail their PCI ASV Scan.

This scenario is a major problem with Vulnerability Management Programs that do not take a proactive approach to security. If solid Minimum Security Baselines were in place and policies which require the application of these MSBs before a system is placed in a production environment also were in place, then a new server would not result in additional vulnerabilities being present in the network. The ultimate goal of security is to help to reduce risk. If your Vulnerability Management Program only identifies vulnerabilities once they are in a production environment, then it will be extremely limited in its overall effectiveness.

How to address this problem:

When vulnerabilities are identified, processes need to be updated to fix not only the vulnerabilities, but also the broken process which resulted in the vulnerabilities. Broken processes could include the patch management process, system hardening guidelines, etc.

You have no solid numbers showing your program is working

It is important to track your vulnerability management program’s metrics. Metrics like the number, severity, and category of vulnerabilities that are identified during your Assessments should be tracked. It also is important to track the total vulnerabilities that have been remediated and the length of time it took to address these vulnerabilities.

Without numbers, you will not be able to perform effective trending analysis; and you will be unable to measure the effectiveness of your vulnerability management program. To clarify my point, imagine having the following conversation with your boss: CIO: How effective is our vulnerability management program? You: I think it’s pretty effective. CIO: Are we more secure than we were this time last year? You: I am not sure, but I just remediated five vulnerabilities this week.

The result of the above conversation may not be so favorable. Further questions like, “Why should I continue our vulnerability management program if we are unable to verify it is working?” or “Why should I approve this budget request if you do not know if this new scanner will help to improve our overall threat posture?” may soon follow.

How to address this problem:

Metrics relating to the organization’s vulnerability management program should be tracked. Metrics such as these should be tracked: the number of vulnerabilities that are identified during security assessments; the severity of these vulnerabilities; the time it took for a vulnerability to be remediated; the category of identified vulnerabilities; the root cause of the vulnerability; the device the vulnerability was found on, and the number of false positives.

The vulnerability scanner gives inconsistent results

You wake up in the morning, take a shower, get dressed, jump in your car, jump on the highway, drive fifteen miles, exit at mile marker 30, make a left and you are at work. Suppose the next morning you did the EXACT same thing: took the exact same highway to the exact same exit, and found yourself at McDonalds? Needless to say, you probably would be a little confused. As humans we expect that if we perform the exact same steps we performed last time, and nothing has changed in our environment, then we should have the same outcome.

This example may sound a bit silly, but in all reality a poor vulnerability management program can be just as frustrating. I have encountered these types of frustrations while performing a vulnerability assessment against devices on a poorly-developed network architecture, or on web application servers which are prone to resource starvation. When performing these assessments, one scan may show that 15 servers are online; a minute later the scanner may show that 17 servers are online; and a third scan shows that only 10 devices are online. In these cases, it is vital to adjust your vulnerability scanners options in order to accommodate these scenarios. Dropping the total number of threads a vulnerability scanner uses from 100 to 10 sometimes is necessary in order to get consistent results.

Another scenario that can lead to inconsistent results is changing the scanners you use to perform your vulnerability scans each month. Your results will be inconsistent if one month you use Acunetix for web application scanning, the next month you use WebInspect, and another month you use AppScan. It is important to be consistent with the tools used to perform Vulnerability Scanning.

How to address this problem:

When inconsistencies are identified, the root cause of the inconsistency must be identified. If the inconsistency stems from the vulnerability management program tools or methodology, adjust these components in order to obtain consistent results. If the inconsistency stems from something that you have no control over and you are unable to adjust something you do control (like the number of concurrent threads the web application scanner uses) in order to obtain consistent results, contact the proper parties in order to address the root cause of the problem.

Many of your “false positives” are actually legitimate vulnerabilities

“Can you mark down this vulnerability as a false positive? Our network team says that they verified SSLv2 is disabled on this device.” This is a conversation I have had many times with our clients after I send them the results of a vulnerability assessment. After I am told that a particular vulnerability is a false positive, I manually check to see if the vulnerability truly is a false positive. Most of the time, the manual retest reveals that the “false positive” is really a vulnerability and the person reviewing the vulnerability assessment results does not understand the vulnerability and / or how to remediate it.

It is important that someone who is a part of the vulnerability management program possesses enough security knowledge that they are able to manually validate the vulnerabilities. When someone claims that a vulnerability has been remediated, it is critical that the vulnerability be manually validated to confirm the vulnerability is truly gone. Unless vulnerabilities are validated, it is possible to have critical vulnerabilities in your network that are never addressed.

How to address this problem:

When someone claims a vulnerability is a false positive, verify that the vulnerability is really a false positive and not really a vulnerability. Never mark a vulnerability as a false positive until you have verified it is.

You count on general-purpose vulnerability scanners to find web application vulnerabilities

When running vulnerability scans, it is important to scan the entire network for general vulnerabilities. It then is important to filter the web applications out of this list and scan them with a separate Web Application Vulnerability Scanner. In 2009, SANS reported that 60 percent of attacks occur on web applications. Because most attacks are focused on web applications, it is more critical than ever to make sure these applications are locked down. Most scanners like Nessus and QualysGuard are limited in their ability to find web application vulnerabilities, and frequently miss glaring web application vulnerabilities, thus the need to use a web application specific vulnerability scanner.

I have found many cross site scripting or SQL injection vulnerabilities that most scanners like Nessus and QualysGuard miss that web application specific vulnerability scanners identify. Examples of some of the more popular web application scanners include Netsparker, Acunetix, WebInspect, and AppScan.

How to address this problem:

Use general purpose scanners to scan your entire network. Pull the web applications out of this list and use a web application specific vulnerability scanner to scan the web application.

Vulnerability scanning is not supplemented with manual assessments

Web application scanners have their place, but in the end they are only one piece to an effective vulnerability management program. In addition to performing vulnerability assessments, organizations also should have regular attack and penetration assessments and manual web application assessments performed. The problem with vulnerability scanners is that they are unable to think like an attacker.

I have participated in external attack and penetration assessments where we were able to leverage vulnerabilities that a vulnerability scanner would not have found, in order to gain domain admin rights on the client’s network. While performing manual Web Application Assessments I have identified web application vulnerabilities which enabled me to run arbitrary commands on the underlying operating system which no web application scanner found. The fact is that web application scanners are limited in their effectiveness, and must be complemented with manual assessments.

How to address this problem:

Supplement automated assessments like vulnerability scanning with manual assessments like Attack and Penetration Assessments and manual Web Application Security Assessments.

No vulnerability management program exists

This one only counts as half because it is pretty much a no brainer. A good Vulnerability Management Program is critical in order to maintain an effective security program. Without a Vulnerability Management Program, critical vulnerabilities could be present in your network and you may not even know they are there. Your Patch Management Program may not be patching half the machines in your network and your Minimum Security Baselines could be missing critical steps which should be applied to devices before they are placed on the network.

The lack of a Vulnerability Management Program ultimately could result in a breach because vulnerabilities attackers use to compromise networks are not being addressed in your environment. Without a Vulnerability Management Program your security team is blind to the technical risk to which your company may be exposed.

How to address this problem:

The solution to this problem sounds simple, but the implementation of it can be a complex process. Simply put, the solution is a Develop a Vulnerability Management Program. The implementation of the solution can be complex in that developing a good Vulnerability Management Program can require much time and effort. Remember that it is better to take a little more time to develop the Vulnerability Management Program correctly in the first place, than to just throw the program together with little or no thought and have to deal with the ramifications of a poorly developed program.

Conclusion

When developing a vulnerability management program, it is vital to ensure the program’s success can be measured; the program is both reactive and proactive; vulnerabilities are being remediated based on pre-determined timelines; and “False Positives” are manually reviewed before categorizing them as such. A poorly implemented Vulnerability Management Program will cause a great deal of headaches and long nights. Although creating a strong vulnerability management program from the ground up requires a lot of work, the rewards can be great for your overall security program.