Patch management – not for the faint of heart

By now, most in the US are aware of the massive data breach at Equifax, with data on an estimated 145 million consumers being disclosed. Initial reports link the breach to a missed patch on a key web software component, which has been available for some time, but not applied by Equifax.

 My consumer side is mad at Equifax. They have been entrusted with my most personal data, and due to their careless patching, I and many other consumers will be looking over our shoulders for years to come. On the other hand, my professional side is inclined to be considerably more understanding. After all, a good patching strategy is very difficult to implement. Even if a leading edge organization finds a way to stay current, they still have trouble keeping up with the zero day vulnerabilities and those that are, as yet, undisclosed. 

Most organizations understand the importance of patching their systems. At the risk of over-simplifying the problem, their reluctance to do so is explained by a simple equation:

patching = downtime = lost revenue

Since many companies have web applications used by consumers at all hours, they never want those applications to be unavailable. Therefore, they are reluctant to schedule downtime for patching. 

The problem is even greater in true 24/7 industries such as healthcare, where patient care at 3am is just as important as it is at 10am. There just is no good time to that systems down to patch them. 

Another challenge lies in the fact that In a typical larger organization, patch management often involves a variety of departments. One group, often Information Security, is pressing other groups to patch their servers aggressively. Those groups often push back, because they don’t want downtime, or have other priorities. Getting the job done in these cases usually involves negotiation and diplomacy, disciplines that we technical folks often find difficult to apply. 

It is clear that patch management involves a number of challenges, but none of us want to be the next Equifax. As such, we must find a way to tackle the problems and secure our systems. The following are some suggestions for achieving a workable patch management program: 

It’s all about risk management

I am a bit of an idealist, so I would love to apply every patch available to all of my systems. Most patches, however, have some impact on the organization. The secret is to balance the need for patching with the need to keep the organization running. Patches for high-risk vulnerabilities on public-facing systems need to be patched quickly, because the cost to the organization of not patching, in these cases, can be more than the cost of the inconvenience. Other vulnerabilities may be less exploitable, only applying to internal systems. These can often be safely delayed. The organization’s risk in not patching must be carefully balanced against the cost of doing so. 

You can’t patch it if you don’t know it exists

It may seem obvious, but you can only patch systems that you know about. Many organizations have hidden systems, which often reside in a closet or dark room somewhere, and are quickly forgotten after installation. An accurate, living inventory of systems is required to have any degree of success with a patch management program. 

It’s not just about PCs and servers

Some of the most vulnerable devices in an organization are not traditional PCs or servers – they are network and Internet of Things (IoT) devices. They too require patching, and most not be forgotten or ignored.  Finally, don’t focus on the OS and ignore software from other vendors, like Adobe, Apache, and many others. 

Patch and vulnerability management go hand-in-hand

A vulnerability management program involves taking  a different view of the patching problem,  by attempting to identify vulnerabilities which require  patching. Using vulnerability scanning systems, such as Qualys, organizations can monitor their systems for actual vulnerabilities, rather than just arbitrarily applying patches. This approach also allows for confirmation that patching did resolve the vulnerabilities for which the patch was intended. 

It you can’t patch, apply mitigating controls

There are legitimate circumstances under which patches just cannot be applied. An example is medical devices, for which each patch must be carefully evaluated and coordinated with the device manufacturer to avoid impact to patient care. This is still an issue for users of many such devices that have yet to have their manufacturers release patches for the WannaCry ransomware worm. In such instances, it is necessary to apply some sort of mitigating controls, such as disabling external network access, or turning off some functionality. 

Be your organization’s United Nations

As I noted above, establishing and maintaining a good patch management strategy takes diplomacy. You will need to give some ground to get some. Look for allies in other teams, and state in clear terms why patching is required, without drama or hyperbole. Strong support from organizational leadership is of great value in getting a patching program implemented. 

Bottom line – patch management is hard. Given technological constraints, this fact is unlikely to change soon. We need to do the best we can to stay current with patching, because the consequences of failure are serious. If you are not convinced, ask a former Equifax executive.