How Visa built its own container security solution

Like many large enterprises, financial services giant Visa has embraced containerization technologies that enable companies to move from legacy monolithic apps to microservice-based application architectures that are easier to maintain, update and deploy at scale on cloud infrastructure. But splitting apps into microservices also comes with the challenge of ensuring the containers hosting the various parts are properly monitored and protected from attacks.

Instead of deploying a combination of commercial solutions and spending resources on getting them to work for its environment, Visa’s security team went back to basics and created its own continuous monitoring solution that handles security policy enforcement, incident detection and remediation, a project that earned the company a CSO50 Award for security excellence. Called MASHUP (Micro-services based Adaptive Security Hardening and Usage Platform), the solution takes advantage of the native capabilities that already exist on container orchestration platforms such as cgroups, filesystem access controls, and SELinux policies, and it is primarily built on top of open-source tools and libraries.

Build vs. buy

Several factors led Visa to create its own security platform rather than go with commercial solutions from established vendors.

For one, many vendors that offer security solutions designed for container-based infrastructure and containerized apps are start-ups, so those products might not yet meet the maturity standards that large organizations expect.

Other products might include the monitoring and protection for containers as part of a much larger feature set that some organizations don’t need. Visa wanted to prevent the kind of feature creep that comes with buying a product of which they would only use 10% the features.

Another big factor in Visa’s build vs. buy decision was development flexibility and agility. Having full control over its platform meant Visa could quickly implement new features requested by internal teams or change the product roadmap based on new priorities and strategies dictated by management. The ability to fix identified bugs quickly was also a factor.

In addition, some of the available commercial products were missing features that the company needed for its specific environment, and the absence of those features would risk leaving threats unmitigated or would have required the company to wait for the vendors to add them.

“Visa decided to build the product because we had significant internal expertise and understood the problems that needed to be addressed,” Sunil Seshadri, CISO for technology and operations at Visa, tells CSO via email. “Furthermore, we found the existing solutions in the marketplace did not fully address our needs. As a result, we purpose built the product to our usage and threat models.”

Finally, building their own solution allowed Visa’s operations, security and development teams to work closer together and support each other. This is the direction that has become increasingly important in recent years with the rise of DevSecOps.

“Instead of bolting on security controls as an afterthought, we were able to make this product as part of the design, lowering the future costs that could have incurred due to unmitigated threats in a live system,” the company said in its CSO50 awards submission. “Visa’s container security product (MASHUP) today has helped Visa deliver critical container and Kubernetes security across the states of build, deployment, and runtime security while protecting its critical application stack running in the Visa private cloud.”

Start small, go big

The company started small, relying on the native capabilities in host operating systems such as Linux, and worked its way from there adding features step by step. Eventually the product became largely platform independent and can easily be adapted to work with any host operating system (OS) or container orchestrator.

MASHUP performs access control enforcement and monitoring at the kernel, SELinux, runtime and container application levels and can distinguish and correlate between host-level and container-level events and activity. It also enforces configurations that are secure by default to prevent loopholes that could be exploited by attackers. This shift-left validation means that only vetted and secure configurations are pushed into production.

A big development focus was on building a machine-learning engine that performs sequential and point anomaly detection by looking for deviations from automatically generated profiles for thousands of workloads that generate millions of events. This was built using open-source libraries such as TensorFlow. If an incident is detected MASHUP responds by applying automated playbooks created by Visa’s security team in order to remediate the situation. According to the company, this reduces incident response time from days to minutes.

“Development of this system took more than two years and we iterated over time. The best part was that K8s [Kubernetes] technology was maturing and iterating rapidly,” Seshadri says. “The same for container technologies. This allowed us to improve the product as new capabilities were realized and new threats identified.”

Scalability and efficiency

The system was built to scale easily and automatically as new nodes and containers are added to clusters. MASHUP works in the background, being completely transparent to the monitored applications. This means that there are no hooks into the application runtimes and no changes required to the application code.

The MASHUP server and agent are packaged as container images themselves, which allows for rolling upgrades with no downtime. Visa said the impact on system resource consumption is small (Visa declined to provide specific performance metrics) and acceptable given all the security functions the system provides and is likely much lower than running multiple commercial solutions to cover the same feature set.

Since the system is now integrated into the continuous integration and continuous delivery (CI/CD) pipeline, security controls are validated in real time and the vulnerability scanning process has moved from a weekly or monthly schedule, which is typical in enterprise environments, to continuous monitoring.

Adoption and results

Over the first year of the project’s development, Visa had already plugged MASHUP into half of its container deployments, and the adoption grew to 70% with full coverage expected by the end of the second year. “From lack of MASHUP as a security control to it being present, [a large majority] of security-related events and attacks were automatically defended with a [quick] mean-time-to-detect … and a mean-time-to-contain of [a matter of minutes],” the company said in its CSO50 submission.

The company believes that it made significant cost savings compared to implementing a commercial solution from a third-party vendor. Money was saved on infrastructure and labor associated with deploying a vendor solution, on fixed and annual maintenance fees, on hiring or training personnel to operate the vendor solution, and on product licenses, which were offset by using open-source technologies.

Can other companies do the same? They should be able to if they have the engineering muscle to embark on such a project and a deep understanding of their environments and data they want to protect. It all starts with a good and systemic threat model.

“While there are similarities between many organizations as it relates to the threat landscape and attack surface, differences exist and are quite meaningful,” Seshadri says. “Some factors organizations should consider include: organizational appetite for building vs. buying, available engineering talent, available operational and engineering excellence, flexibility to iterate over time, having a commercial fallback in case the build does not achieve its goals, and capital, labor, and risk ramifications should the project not succeed.”

In the security world there’s a saying that you should never invent your own crypto algorithms. That’s because there’s a relatively small number of cryptography experts and cryptanalysts around the world who have the knowledge to do it, and their work is heavily scrutinized and subject to peer review before it’s accepted or recommended for widespread use. If we apply the same line of thought to any security system in general, the supporting argument would be that it’s always better to use a system created by experts than it is to build your own.

However, the advances in machine learning over the past five years and the wide availability of free resources and peer-reviewed open-source libraries that allow users to take advantage of this technology are starting to change that. Under the hood, many commercial products use the same open-source tools and native capabilities to capture and analyze data, and they have built statistical models on top of it. Nowadays, those models are well defined inside standard libraries. As long as companies can capture all the events needed to avoid blind spots and know their data and threats very well, they can create good rules-based anomaly detection engines that are better suited for their particular environments.

“Open-source libraries help organizations get off the ground quickly but do not eliminate the other requirements needed to build a successful anomaly detection model,” Seshadri warns. “For example, any intelligent system requires data, a computing platform that can process significant amounts of data, a software stack including machine learning libraries to take advantage of any parameters or trained models, an infrastructure to act and respond to threats in near real-time, and people to execute all of this.”

Visa is evaluating ways it can contribute some of the tools and processes it has developed back to the community.