• United States




6 ways to fight deploy and decay

Apr 11, 20198 mins
Network SecuritySecurity

Even your best security controls will weaken over time after deployment, much to hackers' delight. Take these steps to slow down or get ahead of that decay.

decayed weathered distressed texture security weakened vulnverable
Credit: Mikhail Sedov / Getty Images

Hackers love drift. That’s the unofficial term for describing how some good (and secure) state moves into something less good (and less secure). Computer security is difficult. We know that. The old saw says defenders have to get it all right all the time. Hackers just need to find one mistake.

Any computer control is hard to deploy perfectly. The even bigger problem is that any nearly perfect, deployed control almost always degenerates to a far worse state starting almost immediately. The process is what some security experts call “deploy and decay.”

What is deploy and decay?

For example, suppose your job is deploying and maintaining security settings on an application or operating system. You (or your team) spend a great deal of time deciding what those security settings should be. You read about the negatives and positives of each setting, consider the business impacts, and then choose the setting that best fits the allowed risk profile for your organization.

You deploy those settings, say using Microsoft Windows or Active Directory group policy, to every possible managed computer. We know that for myriad reasons those settings are not perfectly applied to all the computers we expect. It’s usually some issue in the imperfect technology. It could be a corrupt local configuration database, a third-party app getting in the way, or the computer unknowingly not connecting to the domain for months. It can be a lot of different reasons, and we aren’t always aware that the deployed control isn’t applying to all computers.

The same thing applies to patch management. You push out patches and you’re lucky if you get 99% compliance. Often, you’re not sure why, and in a busy environment you don’t always have the time to troubleshoot and resolve. Those of us in the computer security industry, sad to say, have to accept “good enough”. It’s hard to get 100% compliance to any security control out of the gate. Technologies just haven’t invented that animal yet.

Drift also happens because of a lack of change control and putting fires out. Something’s not working on a computer and someone shuts off the host firewall or antivirus software to troubleshoot, and they forget to change back. They move a computer to another Active Directory organizational unit to troubleshoot an issue and forget to move it back. They add a user’s account to a highly elevated group to rule out permission blocking issues. They open a firewall port or use the dreaded, allow ANY-ANY rule, to rule out firewall issues.

In the heat of the moment of trying to solve a mission-critical issues causing immediate pain, it’s easy to forget to write down what you changed and change them back after the troubleshooting fire is over. I can’t tell you how many routers I’ve seen with allowed ANY-ANY rules, there for months and years, because some knucklehead forgot to remove their quick troubleshooting test change. It is normal for security configuration settings to drift over time, and rarely to a more secure place. It’s almost always to a less secure place. That’s decay.

The sheer number of issues and competitive pressures that we face every day in computer security means that some things are going to get less perfect and neglected. Show me any great computer security device and log file that you purchased to improve your security, and if I re-look at it months or years later, it will usually be unwatched, less monitored and more full of useless noise that the staff has learned to neglect.

To make matters worse, the decay is different per device. What is wrong with one device is often completely different than what is decayed on another device. Heck, you get lucky if the bad change is the same on all devices. You’re more likely to notice it sooner and resolve it with one change that applies to the whole environment. What we all face is a combination of subtle global and seemingly random changes that happen across our environment over time. The more time the more drift and decay.

It isn’t just a technological problem. It happens at the human level, too. Social engineering and phishing are responsible for the vast majority of malicious computer attacks. You can provide the best anti-social engineering training in the world to all end-users, and if you don’t reinforce it frequently enough, the people will forget what they learned. Same thing if you sent all your router jockeys to router school or sent all your security employees to CISSP school. Employees come and go. Someone you spent money on to train better will take that training and go work for another employer for more money.

One can argue that deploy and decay is one of the most significant issues we face in computer security. Hackers sure seem to appreciate it.

Solutions to deploy and decay

This is not to say you can’t fight it. There are many ways to minimize the impact of deploy and decay, including these six:

1. Change your culture to be decay-aware

The first step is to recognize that deploy and decay is a normal part of any computer security control environment. Recognize it. Accept it. Communicate it. You want to make your environment aware of its existence and create a culture that fights it. You want to create a culture that takes seriously the risks that decay presents, and as a unit, resolve to fight it to the best of your ability.

2. Budget against decay

Most IT organizations do a great job at budgeting enough money and other resources to deploy a new control or device. Most don’t budget enough resources to the ongoing maintenance and care of that item. Most spend nearly 100% of the resources on the initial deployment, and once done (imperfectly to start with), consider the project a success, and then move on to the next imperfect project.

In many cases, the deployed control has zero time and resources dedicated on an ongoing basis. It just becomes part of the myriad things that everyone is supposed to add to their daily task list, which is a recipe for guaranteed decay. A smart manager realizes that every deployed control and device needs ongoing care and dedicates future resources at the start of the project to the ongoing maintenance of that control/device. If you don’t, decay will happen. Recognize that if you don’t plan resources for the ongoing maintenance of something, you are choosing a future decayed state.

3. Account for decay in your change management policies and procedures

Enforce change management in the culture using policies and procedures. Nothing significant can be changed without prior approval and documentation. Include emergency change management processes and procedures for ad hoc emergency needs. Stuff happens. Good change management is worth its weight in gold.

4. Make troubleshooters put things back to their original state

Make sure all troubleshooters are required to write down all changes they have made, and as part of the resolution process and closing the ticket, put all unneeded changes back to their original states. Any permanent changes need to be updated in configuration setting documentation and change control logs.

5. Automate monitoring and drift resolution

Automate the monitoring and resolution of unapproved changes. If you don’t automate periodically checks on the validity of a control, it will drift and drift faster. How often you check something depends on its criticality. If possible, automate changing back any unapproved setting to its approved state. If the fix back breaks something, update your configuration setting requirements and follow the normal change-management processes.

6. Re-enforce security awareness training

Remember that drift and decay happen in humans, too. You need to reinforce that security awareness training on a periodic basis. Send your new replacement router jockeys to router school to replace the previously trained ones that left for other jobs. Documenting knowledge in an internal wiki just isn’t a replacement for real, hands-on training. Include basic and desired training for every new employee, and refresh what everyone gets on a periodic basis.

Even if you do your best to fight drift and decay, it will happen, but you can slow the rate at which it happens. You might even be able to get ahead of it. Either way, you will decrease your overall cybersecurity risk and make hackers’ jobs harder.


Roger A. Grimes is a contributing editor. Roger holds more than 40 computer certifications and has authored ten books on computer security. He has been fighting malware and malicious hackers since 1987, beginning with disassembling early DOS viruses. He specializes in protecting host computers from hackers and malware, and consults to companies from the Fortune 100 to small businesses. A frequent industry speaker and educator, Roger currently works for KnowBe4 as the Data-Driven Defense Evangelist and is the author of Cryptography Apocalypse.

More from this author