Verizon builds a DevSecOps culture with its developer dashboard

DevOps and the cycle of continuous integration and continuous delivery have become common software development practices. But the integration of development and operations still often leaves security as an afterthought tacked onto the end. This has led to the creation of the DevSecOps concept.

DevSecOps aims to integrate security into every stage of the development processes, meaning everyone is involved in ensuring applications are secure. This can, however, be a hard sell for developers and business leaders who are more interested in results and moving into production. 

Telecom giant Verizon is one such company looking to combine its DevOps practices with a secure-by-design approach. To encourage that culture change, the company created a developer dashboard — a project that earned Verizon a CSO50 award for security innovation.  

Driving culture change

Verizon is the second largest telecom company in the U.S., boasting revenues of $131 billion in 2018 and over 140,000 employees worldwide. Originally formed from the breakup of the AT&T Corporation’s Bell System Corporation, today the company offers a range of telecom hardware, services, and infrastructure to consumers and businesses.

Verizon

As Verizon IT worked to implement DevOps processes and move more applications to the cloud, the AppSec team needed a way to facilitate secure DevOps practices and help drive a culture change within the company. “We needed something that is more sustainable that can help us build a larger influence of our centralized team, and at the same time, not burn the IT application team by keep dumping more work on their to-do list,” explains Manah Khalil, IT director of application security. 

Khalil and his team are responsible for application security across all of Verizon’s businesses and IT portfolios. Khalil’s team also handles security education and awareness among the company’s developers.

To help drive DevSecOps adoption and nurture a security culture, Verizon created the developer dashboard program. It combines technical aspects of vulnerability management with individual accountability to help instill a security mindset among the company’s developers.

Finding insights in developer data

The developer dashboard is a centralized, real-time record of how vulnerabilities are introduced into applications within Verizon’s business. It keeps track of scanning frequency and results, as well as the types and density of vulnerabilities within any one of the 2,100 applications being monitored (measured per 10,000 lines of code). It provides a view of where in the development lifecycle that vulnerability was introduced and by whom.

The dashboard also ties in with learning management systems (LMS) and organizational charts and responsibilities. This provides insight into whether certain teams or individuals are spread too thin across too many applications or projects, are repeatedly making the same kinds of errors, and have had or need more education and training.

“Typically, in a software, you can measure the number of vulnerabilities, you can measure the density, but how does it relate to the culture?” says Khalil. “You have so many dashboards out there; dashboards that are looking at your build quality, at your code quality, how often are you generating new builds, testing it, deploying, etc,” he says. “But those are more meant as a to-do list; the achievements that you’ve done and your technical debt, and by when you can achieve a certain set of milestones and deliverables. We’re trying to use this as a way to look for the change and the culture change.”

The dashboard pulls information from a variety of sources:

• Asset management to provide context around the application’s age and who owns it
• LMS to provide intel on the fields and level of expertise of those responsible for those applications
• Version control and information around technologies used on that application
• Static and dynamic code analysis
• Integrated developer environment (IDE) tools
• Open-source vulnerabilities
• Third-party scanning tools
• Configuration data
• Network vulnerability scanners
• Web and firewall logs
• Computer incident response teams (CIRTs)

“If we want to see the influence of the culture, it has to be quantifiable. If we are injecting a change, we need to see if that change is steering us in the right direction; does it have the intended impact or not. Everything we built in the in the developer dashboard was designed so that we can make a change and then quantify that change.”

How the dashboard improves security

The developer dashboard provides a centralized view of Verizon’s vulnerability risk and gives near real-time feedback to developers on the risks they may be introducing to the business. It also is designed to help create more long-term changes for both the individual and the organization.

“We’re not looking for ‘you still have seven vulnerabilities; you gotta finish them by end of the week’,” says Khalil. “That is not sustainable and is only going to help us in the short term and we haven’t really made any significant change longer term.”

The tool can help highlight why vulnerabilities are being introduced. Perhaps the team is working on a hard-to-update legacy application, or they lack training on a newer programming language or technology. Maybe individuals need training to stop introducing the same types of vulnerabilities repeatedly. If there is no easily identified reason, perhaps the team is not taking security seriously as a priority.

“You’ll see that, perhaps, all the applications managed by the same manager have in general high vulnerability density, and then you can look at the cluster from a technology perspective,” says Khalil. “Is this limited to the .Net or Java or the new Groovy source-code base? Did the training really not get through to a team as I see that only 5 percent of a team has completed their training? Do we not have enough experienced people in the technology, or at the organization level they just haven’t received the email that security is important to us?”

Leveling up on ownership and accountability

While some developers might be wary of a tool that allows superiors to call them out individually for vulnerabilities being introduced into code, Verizon looks at it as a way to increase ownership and opportunity for developers to be better and improve their skills. “We believe that the best way to attack change is at the individual level and bring more accountability,” says Khalil. “You throw a line of code into the repository, you scan it, you find it’s vulnerable, we give you that feedback instantaneously. If you do this repeatedly, you know that we are tracking and then eventually you’re going to ask yourself, ‘What am I doing wrong? How can I get better?’”

There is also a gamification element: Developers can earn “shields” and “medallions” on their profiles for introducing code without vulnerabilities over a certain number of days, while managers can earn them for their teams doing the same.

Khalil says his team has seen seen “a significant increase in the adoption in a relatively short period of time,” and the company has almost all its IT team fully onboarded. “The main reason why we’ve seen the increase in the adoption is that it was not an overhead; we integrated directly with their source control, integrated already with their DevOps pipeline. We have centralized everything. They can see the results and they’re not going to the individual tools, so it’s a positive thing to integrate with us and to adopt the tool.”

The dashboard has also helped encourage executives and management engage more in security. “I’ve never seen as many executive directors and above looking at their own KPIs like we’ve now seen through the developer dashboard,” says Khalil. “When was the last time a senior VP would ask how many vulnerabilities you have, and how long they’ve been in production or in the source repository?”

Designed to be open

The project is around two-and-a-half years old, and initial development took around six months to build. The dashboard is powered mainly through APIs and connectors that turn all the disparate data sources into a generic format that the dashboard can understand, with the aim of being flexible and open to the addition of further tools in the future.

“The model is pretty generic in the repository, and then you have these adapters that can connect to a source and then translate from their native format to our format,” says Khalil. “So, rather than go put more work into coding specifically for tool A vs. tool B, we built these adapters in between. When we consume the data, we do the conversion into our own format assuming that in the future we may need a specific tool because the existing one doesn’t cover a specific technology.”

Verizon plans to release the developer dashboard to the open-source community in the future. Khalil says the codebase is stable and April is the current goal for releasing the project to the community. It will include adaptors for the tools Verizon uses internally and connectors to other tools during the dashboard’s proof-of-concept development.

“There are still several data points that we want to integrate with, around the privacy and protection of the customer data,” says Kahlil. “We want to see if we can potentially link some of the incident response [data] so you can have a better oversight of the system.”