SolarWinds: “IT’s Pearl Harbor”

What do SolarWinds, Fidelis, FireEye, Microsoft, Mimecast, Palo Alto Networks, and Qualys all have in common? Each and every one were victims of the SolarWinds software supply chain attack. There are more, many more. The Russian government’s hack of SolarWinds’s proprietary software, Orion network monitoring program, ruined top government agencies’ and tech companies’ security. Months after it was first revealed, we’re still trying to get our arms around just how bad the breach was.

That’s in no small part because while the Orion platform exploits were more than bad enough, the hacker group, widely believed to be Russian government-affiliated Cozy Bear, also used that breach to distribute malware such as Sunburst. Other programs, such as Microsoft Office 365 and Active Directory, were also used in the global attacks. Indeed, Brandon Wales, acting director of the US Cybersecurity and Infrastructure Security Agency (CISA), told the Wall Street Journal (WSJ) that almost a third of the victims hadn’t been running SolarWinds Orion at all.  It’s the security breach that just keeps on breaching.

As bad as bad gets

You don’t have to take my word for it. Azeem Aleem, NTT’s Global Digital Forensics and Incident Response Lead, said, “There is no end to the fallout in sight. In addition, organizations must consider that more threat actors are likely to mimic the SolarWinds incident, given the success of the attack.” Indeed, the FBI believes a Chinese hacking group has already taken advantage of another SolarWinds security hole to launch attacks.

So how bad is it? Tom Kelly, president and CEO of IDX, a provider of identity protection and digital privacy services, fears that, “On a scale of 1-10, the SolarWinds hack was definitely a 10. It however could even turn out to be a “bigger 10″ before the dust settles. We are likely to learn more about its scope and impact over the next few months.”

Kelly’s far from the only security expert who thinks we still don’t appreciate just how bad this attack has been. Jonathan Moore, CTO of the cybersecurity firm, SpiderOak, said, “It’s much bigger than the headlines make it appear. While hackers used the SolarWinds Orion product to gain a foothold on networks, penetration deeper into victim networks was done by hand with operators at keyboards. These attacks likely involved hundreds, and potentially thousands, of work hours. At this point, the firms investigating the attack have reported that the Russian group actively breached upwards of 300 organizations, with each compromise custom-tailored and hand-executed against the target.”

That’s not all. Moore continued, “The public, however, is missing the true scope of this attack. The security crisis we are in has been driven not by one individual company such as SolarWinds, but by negative market incentives that ultimately lead to breaches. From a business perspective, it is not possible to analyze the risk that software products introduce, and security products cannot quantify the reduction in risk they provide. Customers are therefore motivated to spend resources on the portions of their business where risk and rewards are better understood, meeting only the minimum bar in security.”

Kelly added, “The SolarWinds breach has numerous factors that contributed to its extreme seriousness. It used the trusted relationship that organizations have with their IT supply chain, in this case, SolarWinds, in order to avoid detection and broadly distribute the attack. Additionally, it relied on a technical approach that was able to circumvent detection by government cybersecurity efforts. Lastly, it took a very patient approach to exfiltrating information in order to reside and operate undetected for many months in organizations’ networks.”

Pete Slade, CTO & Chief Scientist at ThreatWarrior, a security company specializing in network and supply-chain threat detection, said “The SolarWinds hack could definitely be thought of as ‘IT’s Pearl Harbor.’”

The worst may be yet to come.

The worst was still to come. Unlike most programs, which attack immediately and cause visible damage, Slade continued, “The malware lay dormant for two weeks, evaluating its environment and blending in to evade detection. This attack was impressive in its approach and scale, and we’re still learning about its extent even beyond SolarWinds. CISOs everywhere have now woken up to the fact that they can no longer trust their security supply chain, and they’re looking for a solution to that problem. It’s a grim realization that trust itself has become the attack surface and our most trusted solutions require the same level of scrutiny as our least.”

Karen Walsh, cybersecurity compliance expert and Allegro Solutions CEO worries, “The SolarWinds security breach is the Three Mile Island of cybersecurity because it highlights the long term consequences.” Walsh explained, “It’s important to understand the difference between “Sunburst” and “Supernova.” Supernova was a malware installed using a vulnerability in the Orion platform, not a malicious code embedded within the builds. The real concern is Sunburst.”

Why? Walsh explained, the January 29th SolarWinds Security Advisory for Sunburst said the “cyberattack to our systems that inserted a vulnerability (SUNBURST) within our Orion Platform software builds.” Walsh continued, “It’s important to emphasize that the vulnerability was inserted within the builds because that implies manual touches that compromised the product itself, not just a code that you end up downloading to your own devices or systems. Sunburst is analogous to the Three Mile Island station meltdown, a single disastrous event. Supernova is the continued ecosystem contamination that comes afterward. Because Supernova’s certificates looked legitimate, the supply stream contamination seeped into the ecosystem undetected.”

Walsh continued, “Finally, to conclude the analogy, you will have organizations finding indications of compromise for a long time, never really knowing whether it was from the SolarWinds hack or something else. A good example of this would be the alert from Microsoft that someone had viewed the code but not changed it. Three to five years from now, an organization might notice something similar but not be able to confidently attribute it to Sunburst and Supernova. This, then, is similar to the Three Mile Island cancer clusters. We will be tracking the cancerous outcomes of the SolarWinds hack for years, possibly over a decade. Many analysts think that we won’t even realize the full impact for at least five years. So, while it’s tempting to argue that this is ‘another example of poor security,’ this particular hack is far more insidious and sophisticated.”

As bad as it is, Patrick Kelley, Critical Path Security’s CTO worries this attack was just the tip of the iceberg. “This attack hasn’t been as significant as it really could have been. I personally feel that this was testing of the waters. Future attacks on supply chains will include ransomware and larger impacts. Imagine if 18,000 government agencies were hit with ransomware at the exact same time. That is the direction that I really believe this is going.” Isn’t that a thought to make you feel all warm and cozy on a cold winter’s night?

But, wait there’s more. Sachin Bansal, the General Counsel at SecurityScorecard, observed, “The scariest is what we don’t know about this attack (e.g., impact to government systems and/or critical infrastructure), and we won’t know that for a while, and that should be very concerning.”

Bansai added, “This is a supply chain attack, and the concerns about software and hardware supply chain that some may have viewed as theoretical have now been put in very real and stark terms. What other supply chain attacks have occurred since SolarWinds, i.e., how many other SolarWinds are out there? SolarWinds is like a new pandemic — a contagion incident that is evolving.”

Another way of looking at the breach, and just as disturbing in its own way was expressed by  Dr. David Brumley, Professor at Carnegie Mellon University (CMU) and co-founder and CEO of ForAllSecure “We need to acknowledge SolarWinds is the equivalent of a high-class cat burglar, a Moriarty, or an Arsene Lupin. The SolarWinds attack was methodical, careful, and well-done. It’s a sign that nation-states are investing tremendous resources into cyberattacks, and it’s working.”

Oh, and by the way, Brumley continued, “There isn’t going to be a silver bullet. No one did anything ‘wrong’ here. Sure, processes can be improved. We can get better. But this doesn’t strike me as negligence or a lack of security. Sometimes the attacker just wins, at least temporarily.”

Besides the actual damage caused by the attacks, cloud data integration and data integrity Talend’s CISO Anne Hardy, observed, this is a “real wakeup call for taking cybersecurity action. It’s raised large concerns for CISOs and other security professionals – mainly, a lack of visibility and understanding of the vulnerabilities among our suppliers. The SolarWinds hack caused a disruptive violation of trust between vendors. I am concerned with the lack of visibility down our supply chain. We can manage our vendors, but how do they manage their own suppliers? Several companies were indirectly affected because their suppliers were themselves directly or indirectly affected. Until the SolarWinds case, most companies did not ask their vendors about their own suppliers. This will clearly change.”

That said, Rajesh Khazanchi, co-founder and EVP of ColorTokens, a zero trust security company, observed in Sonatype’s 2020 State of the Software Supply Chain report the “next-generation supply chain attacks have surged by 430% in the past year. As the adversaries are getting craftier and imposing an unprecedented level of risks, the world should look to specialized coverage indispensable to cybersecurity protection against evolving cyber-attacks.”

Indeed, it’s not like software supply chain security wasn’t a known issue. As Prakash Linga, software supply chain company BluBracket’s CEO said when the company was launched in 2020–ironically about the same time SolarWinds software was being opened up like an oyster–CIOs, CTOs, and  CISOs often can’t answer such simple questions as where is their code, who has access to it, and where did it come from?

Jim Zemlin, the Linux Foundation executive director and BluBracket board member said, we’ve “seen traditional models and tools struggle to keep up with the pace set by developers and DevOps. Code security that respects developers’ productivity is a critical need for companies who see software as the foundation of their competitive advantage.”

Protecting yourself from software supply-chain attacks.

How do you do that? The Linux Foundation has some ideas on how to secure the software supply chain while keeping today’s hyper-fast continuous integration/continuous deployment (CI/CD) pace. First, David A. Wheeler, the Linux Foundation’s Director of Open Source Supply Chain Security, explained that in the Orion attack that the malicious code was added by subverting the program’s build environment.

This neatly defeated the usual standard security advice:

• “Only install signed versions” doesn’t help because this software was signed.
• “Update your software to the latest version” doesn’t help because the updated software was the subverted one.
• “Monitor software behavior” eventually detected the problem, but the attack was quite stealthy and was only detected after tremendous damage was done.
• “Review source code” is not a certain defense either. In Orion’s case, it’s not even certain that developers could have spotted the source code changes. The changes were carefully written to look like the expected code. In addition, since the attackers had control of the build environment, they could have inserted the attack without it being visible to software developers.

So, what’s the fix? In the long run, Wheeler thinks there’s only one true answer for this kind of attack: Verified reproducible builds. These are ones “that always produces the same outputs given the same inputs so that the build results can be verified. A verified reproducible build is a process where independent organizations produce a build from source code and verify that the built results come from the claimed source code.”

Lovely idea, and we’ll get there eventually, but today almost no programs are verifiable. The Linux Foundation and Civil Infrastructure Platform has been funding work, including the Reproducible Builds project, to make verified reproducible builds achievable.

In the meantime, Khazanchi suggests you use the following mitigation strategies to defend against further risks from this and other sophisticated cyber attacks.

• Ring-fence third-party servers and internal critical applications to prevent unauthorized communications between systems and reduce propagation via lateral movement.
• Ensure your endpoints are protected to prevent hackers from launching legitimate applications and processes from within malicious code.
• Implement continuous monitoring security practices that look for attack patterns exploiting trusted processes and prevent further connections and beacons.

Khazanchi concluded, “Applying security hygiene and east-west segmentation along with endpoint and server hardening can be effective techniques to reign in spiraling complex segments that promote unseen lateral movement. Such attacks depend on network complexity and lack of east-west controls to move laterally from system to system. Micro-segmentation that automatically prevents communication between systems that do not otherwise communicate significantly reduces the propagation possible via lateral movement.”

Finally, using open-source software can be helpful. As Eric S. Raymond, an open source founder, famously described ‘Linus’s Law’ as “Given enough eyeballs, all bugs are shallow.” Open source isn’t a cure-all by any means. But, at least you can see for yourself the code, which might be causing you trouble if you’re worried.

There are programs such as Red Hat‘s Release Monitoring, nvchecker, or Replogy, which can help you spot what’s new in your open-source software. Other third-party code analysis programs Synopsys’s Black Duck or Sonatype Nexus Lifecycle can help.

Another Linux Foundation project, the Open Source Security Foundation (OpenSSF), combines forces with the Core Infrastructure Initiative (CII), GitHub’s Open Source Security Coalition, and open-source security-conscious companies Its goal, said Mark Russinovich, Microsoft Azure’s CTO is to help developers better understand the security threats that exist in the open-source software ecosystem.

The Foundation’s four goals are: 1) Help developers to spot security problems, 2) Provide the best security tools for open source developers, 3) Give them best practice recommendations; and 4) Create an open-source software ecosystem where the time to fix a vulnerability and deploy that fix across the ecosystem is measured in minutes, not months.

These are all works in progress. In the meantime, all we can do is to keep our eyes on our software supply chains. Few of us are used to doing that, but after SolarWinds, it’s clear that we must do so. As so many experts said, as bad as it’s been, it could have been much worse. Indeed, it may yet prove to be much worse.

Let’s take care and be safe out there.