Does DevSecOps eliminate the segregation of duties between security and DevOps?

I spoke to a number of attendees at RSA 2018 about their efforts to build DevSecOps teams, and most believe their organizations’ cultures can embrace the fusion of security and DevOps. That’s encouraging as more enterprises adopt containers to enable their DevOps team to build and ship cloud-native applications. However, that does not mean you should relinquish security responsibilities to the DevOps team. You must still oversee risk evaluation, create and enforce policies, and direct incident response.

I was not surprised to hear more RSA attendees say they are involving DevOps in security processes. We’re seeing more openness on the part of DevOps teams to get involved with security, and vice versa. The perception that Security too often puts up roadblocks to prevent them from getting more done in less time may still linger with DevOps, and Security may view DevOps as caring more about speed than security. But setting aside the old “Us vs. Them” mentality is critical now that containers have moved past the experimental phase, where early adopters used them only to build betas and proofs-of-concept. Today, enterprises are running multiple mission critical apps on containers.

The fact that all parties can embrace the business benefits containers offer is driving this change of attitudes. DevOps and the senior business leadership team like containers because they improve agility and enable faster shipping of applications. Information security teams like containers because they are more secure than traditional DevOps environments. But all parties must be aware of the new vulnerabilities containers create.

My colleague Liz Rice in her keynote address at last month’s KubeCon 2018 entitled “Running with Scissors” walked attendees through how easy it can be for a Docker container user to gain “root” privileges inside the entire node. KubeCon is an event for DevOps, and her presentation speaks to them. But even if you’ve never written one line of code, you should find it both interesting and alarming to see how a user can exploit a (now fixed) vulnerability to escape the container and become a root on the host server. There will others like that in the future.

DevOps use containers every day, so it’s critical that your effort to secure those environments includes DevOps’s input from Day 1. However, there are two primary reasons why DevOps should not “own” security.

First, it’s not their job. DevOps’ priority is speed – faster application development and deployment. If you also make DevOps responsible for security, you create a conflict of interest. Inevitably, a developer will need to decide between deploying an application that he believes may be vulnerable or halt the production process to deal with the security issue. That puts him in an impossible position.

Second, you need to maintain the segregation of duties between your information security team members and DevOps. It’s just like every other piece of the enterprise security structure. Security is responsible for database security, not the database administrator. You own network security, not the network administrator. 

The same applies to DevOps. Security is responsible for securing containers, not the developers working within them. Yes, tools like Kubernetes have security features, and DevOps may argue that requires placing security in the hands of the administrator who runs the Kubernetes cluster. Don’t concede that point, security must maintain oversight and control.

That’s not to say you should not empower DevOps to detect and fix issues. That increases the likelihood of catching vulnerabilities earlier in the development process. But there still has to be the figurative adult in the room who creates the security policies and monitors to ensure adherence to those policies.

The same applies if your organization needs to monitor for compliance. That is not the responsibility of the security or DevOps teams. You should involve your compliance or risk management team in your DevSecOps initiative.

Once you have established a clear segregation of duties, you can address two additional key priorities to ensure DevSecOps readiness. The first is applying security across the entire application lifecycle. If you look back at the history of application development, you see that security was too often an afterthought, even as organizations devoted more information security resources and budgets to securing their networks and endpoints. As I addressed in an earlier article, ”Left-shifting enterprise appsec: what we can learn from mobile app developers,” you need to embrace the developer “Shift Left” mindset to secure containers before they move into production and throughout the dev cycle.

The second priority is automating application security controls. You cannot implement Old World manual processes that impose slow, controlled steps that drag release cycles out for several months. Again, you want to enable developer agility and help your organization achieve its business goals.

You have to automate in more than one place, such as enabling automatic testing as developers move through their applications through the pipeline leading up to deployment, and incident response. In cloud native environments, DevOps may be running clusters with hundreds or even thousands of nodes, so identifying and containing attacks via manual processes is not feasible. You cannot keep track of what’s happening on every single user console, and by time the user figures out what has happened and initiates a response, the attack may have already spread across other nodes, applications, and even to other clouds.

A DevSecOps team that brings Security and DevOps together, yet also maintains clear segregation of duties and responsibilities, will enable your organization to develop and ship cloud native applications faster than ever, and still maintain a strong enterprise security posture. However, this needs to be more than a handshake agreement where DevOps confirms they understand the need to prioritize security and follow policies and procedures you establish. It’s important to formalize DevSecOps, and what form that takes will differ from one organization to another. For instance, you may create a joint team of security and DevOps personnel, or permanently embed security personnel with DevOps.

The goal is to set very clear boundaries for who does what, and who owns the process overall. Failure to do so creates security gaps. In the best case, these gaps emerge as vulnerabilities late in the development process and cause delays. Worst case scenario, no one discovers a vulnerability until it’s too late and you’re hit by a data breach.

Do not leave this to chance. The rash of massive data breaches that have made news headlines over the last few years typically are not due to a single point of failure. An attacker may have taken advantage of a vulnerability, but more often than not, there were steps the organization could have taken to prevent the breach from occurring, or at least detecting it earlier and mitigating the damage. A formal DevSecOps team establishes security controls and processes for conducting audits and penetration testing to help you avoid systemic failures that result in a breach or other successful malware attack.