5 tips for getting started with DevSecOps

At an increasing number of organizations, software development has been undergoing waves of change: First agile development, then DevOps and now secure DevOps, a.k.a. DevSecOps. The net gain is better quality applications that aren’t moonshots in their first iterations; instead they launch quicker and evolve into greatness.

But the advantages of the DevOps methodology haven’t always been fully realized because of the elephant all too frequently not in the room: Security.

With the security waterfront becoming a higher priority for many, if not most, enterprises over the past few years, DevSecOps’ time has come. According to last year’s DigiCert survey, “Making Security Agile,” 88 percent of those polled said it is important to integrate security into DevOps. The report noted the top three dangers of not inviting security to the DevOps table: additional security risks, increased costs and longer delivery cycles.

While DevOps has been largely successful for many companies, the initial process didn’t emphasize the security team’s involvement — indeed, at many companies there are deep cultural and organizational divides between security and development.

The tips and pointers to follow will be invaluable to CISOs and other security professionals who are learning how to champion DevSecOps at their organizations. Because let’s face it: It’s becoming imperative that the security team integrate with the DevOps process.

DevSecOps is part of an evolution

Before there was DevSecOps, there was DevOps. While this may be apparent to most, it warrants calling out because of the amount of hype and hoopla DevSecOps is currently receiving. Secure DevOps is an evolutionary wrinkle of DevOps — one that should have been included from the start.

At its heart, DevOps is about changing the software development process to make it continuously iterative by automating integration, testing and delivery processes. This involves changing company culture and expectations of the software development team. It boils down to this: trying for perfection in a single leap takes too long and rarely succeeds anyway. So, a key principle of DevOps is continuous integration/continuous delivery (CI/CD) of code. A continuous development process that works with automated testing and feedback is proving to be much more efficient for many organizations. And it will be more efficient yet if security is baked in from the start. If you are starting from scratch, adopt DevOps and DevSecOps at the same time if possible.

Some DevOps practitioners consulted for this story are uncomfortable with the term DevSecOps because it implies that it’s something new and separate — when it’s more akin to a moniker like DevOps 2.0. Whatever you call it, you can’t implement DevSecOps without DevOps, which in turn is a natural outgrowth of agile.

1. Plan for culture change

At Fannie Mae, VP and CISO Chris Porter, who led his organization’s successful DevSecOps transition, points to cultural and organizational changes as key to implementing DevSecOps. (Fannie Mae garnered a CSO50 award on its move to DevSecOps.) In fact, most of the experts interviewed for this story were in lockstep with Porter on this issue.

Your primary goal is to foster a culture in which security is a shared responsibility across Development, Security and Operations. That’s going to require some changes in thinking in all three departments. The cultural aspect needs a game plan all its own. Here are some steps that may help CSOs/CISOs champion the process:

• Learn the ins and outs of DevOps and DevSecOps and prepare your talking points.
• Partner with your VP of product and VP of engineering or their equivalents
• Consider hosting a cross-functional meeting to review the security challenges in the framework of the development and operations process, suggest Jimmy Pham and Haluk Saker, modern software development capability leads at Booz Allen Hamilton.
• The security team may face some of the biggest challenges in moving to DevSecOps. Get them onboard early. One of the lessons of Fannie Mae’s transition to DevSecOps is that the InfoSec team has moved into more advisory or consultative role as software engineers take more responsibility for securing their code. That’s a good thing, but the cultural change may be difficult for some security employees.
• DevSecOps relies entirely on collaboration; fostering it as a top priority. It may be your biggest challenge initially.
• Traditional security cultures have been quick to say “no,” often for good reason, because the security review came late in the process. “The biggest cultural challenge to DevSecOps” said Eric Johnson, application and security curriculum manager at SANS Institute, is letting go the “culture of no” in favor of “working together to solve problems, empower teams, fail fast and continuously improve.” It’s usually incumbent upon CSOs/CISOs to lead the charge on this. When security team members must raise the red flag, they should try to suggest alternatives and be part of the solution.
• Use sound change management practices to support employees during the transition to DevSecOps.
• Reorganization may be necessary, as may hiring new employees with DevOps experience.

2. Remap the process

“From an organizational perspective, DevSecOps requires security and compliance teams to integrate their processes, policies and requirements into the same [DevOps] workflows being used by Development and Operations,” Johnson said. DevSecOps won’t work, he adds, if the Security team lacks a good understanding of development and operations engineering processes, technologies and tools.

One of the most important tenets of DevSecOps and a key process change is “shifting security to the left,” which simply means that security analysis and testing must occur much earlier in the process. Security team members should have a seat at the table of every single project from day one, helping to take responsibility for the success of the project. Developers will need to learn coding for security. The security team may be able to help with that, but software engineers are likely to need training.

No one process fits all companies. Booz Allen recommends “journey roadmaps, which simulate everything and anything a user goes through for a specific function.” This is one way to establish what your final process needs to be able to handle and accomplish.

3. Automate for efficiency and accuracy

Automation is a central theme of DevOps, and that continues in DevSecOps. That plays into another key mantra of DevSecOps: Security as code. You’ll want to automate the security testing and as many other touch points as possible, create secure shared code repositories for software engineers to pull from and streamline the security processes to mesh with the speed of the DevOps CI/CD pipeline. Making this work will be a collaborative effort of Security, Engineering and Operations. It’s a key aspect of the transition to DevSecOps, so put some good people on it.

4. Choose software tools wisely

One of your chief considerations in choosing tools should be how easily and effectively you can connect them to DevOps’ CI/CD pipeline and other pre-existing tools. Another is the functionality that each tool delivers and how well it can be automated.

SANS Institute put together a PDF that shows a wide spectrum of DevOps and DevSecOps tools organized by application called the Secure DevOps Toolchain and SWAT Checklist (PDF download).

Johnson selected data from the Checklist for being exemplary of both the security touch points most organizations will need to implement within the DevOps process as well as some of the tools you might consider for them:

• Pre-Commit Security Hooks — code checkers to scan for secrets (private keys, system passwords, cloud access keys, etc.) before committing code to version control.
• Pre-Commit Code Scanning — code editors such as Atom, Code, IntelliJ, and Visual Studio have security plugins that allow lightweight security checks to be run as code is written by engineers.
• Manual Security Peer Review — code review tools such as GitHub, GitLab, and Bitbucket have built in features for performing security reviews on high-risk code changes.
• Static Source Code Scanning — scanning source code files (infrastructure templates, application source code) for security vulnerabilities in the build pipeline.
• Security Unit Tests — custom security unit tests written by the security and engineering teams to provide coverage for abuser stories and negative test cases.
• Dependency Management Scans — scanning open source infrastructure templates and application components for known vulnerabilities.
• Container Security Scanning — With the container (e.g., Docker) phenomenon continuing to explode in the DevOps world, security teams must analyze container images for vulnerable packages, exposed secrets, insecure configuration, and policy compliance.
• Dynamic Application Security Testing — Scanning a running application for common security vulnerabilities (OWASP Top 10) and misconfigurations and enforcing acceptance criterial using tools such as Ganutlt / BDD Security.
• Infrastructure Acceptance Tests — Infrastructure acceptance testing can verify gold images against hardening baselines (e.g., CIS) using tools such as InSpec and OpenSCAP.
• Secrets Management — Automating the provisioning and storage of secrets using tools such as Vault, Conjure, AWS KMS, Azure Key Vault.
• Continuous Monitoring — Monitoring the production environment for vulnerabilities, anomalies, and in progress attacks using tools such as statsd, graphite, graphana, etc.

5. Measuring success

It’s important to put metrics in place in advance of a major project like this. It’s the only way to truly learn whether the payback is worth the investment. The somewhat surprising aspect is that there are no tools or metrics currently available to help you with this measurement. What organizations are adopting, according to Booz Allen and echoed by others, are based on these data points: lead time to deliver, the number and frequency of deployments, the mean time to recover and variations in the number of defects expressed as a percentage. Some automated code quality scanning solutions give a security code quality score that can also be used as a metric. A combination of these data points is probably your best yardstick.

DevSecOps resources

Brush up on various aspects of DevOps and DevSecOps with these resources:

• DevOpsSec ebook – Jim Byrd
• Agile Application Security – Laura Bell, Michael Brunton-Spall, Rich Smith and Jim Bird
• DevOps Culture (Part 1) – John Willis
• Securing DevOps, Safe services in the Cloud – Julien Vehent.
• Securing the Continuous Integration Continuous Deployment (CICD) Pipeline – Fabian Lim
• Making Security Agile – DigiCert survey

Last word

DevSecOps is a worthy undertaking that can have significant payoffs for your enterprise. If you are a DevOps house already, it’s a must. Even if you’re not, although the project becomes much bigger, the move to DevSecOps is well worth your consideration as it’s one of the key underpinnings of digital transformation.