DevOps at its heart leads to continual improvement in software through continuous deployment: develop, deploy, and repeat. But continuous deployment requires lots of changes to that software on a weekly or even daily basis. As you well know, change can be the enemy of security.
So how do we maintain security in software created in continuous deployment environments?
Security challenges & solutions in continuously deployed software
Hackers frequently target specific vulnerabilities that allow cross site scripting and SQL injection. Defensive programming seals these holes using input validation, where you check all input into an application and reject any harmful input. “You simply don’t allow input sequences that could be used to inject code into a browser or database,” says Tom Stiehm, CTO, Coveros.
[ Software security for developers ]
“For SQL injection, a great way to do defensive programming is to only use prepared statements and not allow the use of dynamic SQL. Given that a lot of the most popular and successful attack vectors are injection attacks, input validation is an excellent play to start with defensive programming techniques,” says Stiehm.
There are built-in .NET libraries that help with input validation on that platform. “With Java, the tools include the OWASP ESAPI library, which has security controls including input validation,” says Stiehm.
While input validation helps with programming applications, you need monitoring to see any changes in underlying systems and components that are using the applications you’re developing. Tools such as Chef or Puppet are useful in comparing changes you make. Keep track of whether you have introduced a security flaw into the software through a scripted update or a third party component. “You need to be able to fix it as soon as you discover it,” says Chris Raethke, CTO & co-founder, Bugcrowd.
Tom Stiehm, CTO, Coveros
If you know who on your team owns each set of issues, you can flag any concerns from failed tests for them and stay on top of those and run through continuous deployment processes successfully, according to Raethke.
[ Why bug bounty hunters love the thrill of the chase ]
Tools and approaches such as automated secure code review and penetration testing can help spot and flag issues, ensuring that development teams find and fix vulnerabilities. Code review tools include Cigital SecureAssist, an IDE plugin for real-time vulnerability detection and HP Fortify Static Code Analyzer, for example.
Pen test tools include the Kali Linux penetration testing Linux distribution and the OWASP Zed Attack Proxy. Developers must use whatever tools they select religiously and respond to every vulnerability in order for these to work.
Let The Inhumans Lead
A most powerful element of DevOps approaches is that they increasingly remove the human being from the equation, including the human error, through automation. “But repeatability doesn’t necessarily give you quality,” says Scott Morrison, senior vice president and distinguished engineer, CA Technologies.
In fact, the biggest flaw in automated development processes is in building in security vulnerabilities at the start and letting the assumed assurances of repeatability lull you into complacency. If you don’t fix or remove those flaws at the start, they will continue in each iteration of the software.
To preempt hidden, repeated security vulnerabilities, start with a common security layer that many systems can share in and make that a cornerstone, says Morrison. The common security layer can start with a default Linux OS image, for example.
If you are working in a cloud environment and you spin up the same base image for your applications every time, make sure the system in that image is hardened. At least in this way, the base image won’t be the source of any repeated security vulnerabilities.
Otherwise, when you overlay your application code and as you repeat that many times with the open vulnerabilities such as unnecessarily open ports and services that you should have turned off, you could be creating a vulnerability that you don’t catch now or later, and all systems will share the same problem, says Morrison.
Next, ensure that all communications between different tiers of applications are using APIs that you have built in a secure manner. Apply this as a common baseline level of security. “When you build any distributed system, use APIs to communicate between them. If you get that piece of communications right, then you will have a good solid foundation for security,” says Morrison.
Finally, make it impossible for development to change anything that should never change, such as any security elements that were done right the first time. These should always remain the same. “If you secure baseline images, communications, and applications in these ways, you should be able to enjoy the benefits of repeatability with peace of mind,” says Morrison.
Security Never Sleeps
“Security never stops; it’s a moving target. You have to be vigilant. The bad guys are getting better at what they do at an extraordinary pace with zero-day attacks and other threats popping up everywhere,” says Morrison. You have to stop and analyze each new threat that affects your business. Determine whether you have to apply patches or updates or make changes to any systems or system elements whether applications, operating systems, or APIs.
The goal of DevOps and continuous deployment are automation. Yet, when a security issue arises that automation does not currently address, you may have to go in manually to fix the vulnerability as a part of being proactive. “Even if we reach 100-percent automation, we will always see an eventual interruption somewhere,” says Morrison.
Sometimes consciously thinking outside the box means manually working apart from automation. If you’re on top of security issues and their causes, you’ll know when you have to use manual fixes.
But, be careful that these one-off manual operations to fix security don’t turn into opportunities where human error enters in and creates other issues. By breaking out of the DevOps rhythm, going inside, and being responsive you may also inject chaos into the system, according to Morrison. Test and retest after these operations to confirm you didn’t do this.
Then step back and make sure that you feed these positive changes to shore up security back into the continuous deployment process from the very beginning so it becomes a permanent, repeatable part of the automated software changes, explains Morrison.
In addition to automation, DevOps is about rolling with the punches and dealing with the realities of constant change. “Good DevOps will still have a human element, but will also have strong enough processes to constantly audit and do bookkeeping about the human element,” says Morrison.
Security Cultures
Continuous deployment enables transparency, a view into the software and the changes that developers make. This is a good thing. But the challenge lies in having enough eyes to leverage that transparency and catch mistakes before they go live, says Raethke.
By growing a culture of security among continuous deployment team members and stakeholders at the outset, the enterprise can become more secure over time rather than breed incestuous software vulnerabilities that sprawl across everything the company produces.