5 questions CISOs should be able to answer about software supply chain attacks

Back in December, the world learned about one of the biggest and most sophisticated cyberespionage attacks to date that involved state-sponsored Russian hackers breaking into the networks of US federal agencies and numerous companies. The attackers compromised their victims by injecting malicious code into the legitimate software updates for a popular network management platform developed by a company called SolarWinds.

Several months later, the US government and private industry are still working on uncovering the full scope of the attack, but the incident has brought widespread attention to an issue that security researchers have been warning about for years: the security of the software supply chain.

As companies scramble to investigate whether their own systems and data were potentially impacted by the SolarWinds compromise, executives, boards, and customers are discovering that the threat of supply chain attacks expands beyond this one single incident and that mitigating the risks associated with them is not straightforward. Here’s what security leaders and experts say are the most important questions CISOs need to be able to answer following a software suppy chain breach like SolarWinds.

1. Are we at risk even if we’re not using the backdoored software?

After an attack like SolarWinds happens, business leaders will and should ask IT and cybersecurity managers whether their organization directly uses the impacted software. If the answer is yes, the company’s security incident response plan will be triggered to identify, contain, and remove the threat and establish the extent of the impact to the business.

If the answer is no, it doesn’t mean the organization is safe. The follow-up question should be: Have any of our partners, contractors, or suppliers been compromised? Supply chain attacks can have a wide reach and companies regularly provide other parties with access to their data or networks and servers.

“Our customers sent out some kind of risk management or risk disclosure form and I had to fill this out,” Jesse Webb, technology and security officer at healthcare IT company Avalon Healthcare Solutions, tells CSO. Avalon wasn’t using the software, but “almost all my health plans reached out to me immediately and asked: Are you using it, and if you are using it, stop using it and inform us of your investigation. I did the same thing. I went down to some of my key third-party providers and only had one provider in my supply chain that actually used SolarWinds. We’ve been working with them to pay attention to their forensic response.”

It’s critical to understand that software supply chain attacks are highly complex and can be far reaching over time. A Chinese cyberespionage group known as Winnti, Barium or APT41 has been engaging in this kind of attack since 2017 when it backdoored server management software produced by a company called NetSarang. That same year the group managed to backdoor popular system optimization tool CCleaner and delivered the malicious update to over 2 million computers, though only a select number of companies from the technology sector were targeted with the second-stage payload. In 2019, the group backdoored the ASUS Live Update Utility and served it from the official ASUS servers. Researchers believe all these attacks could be interconnected, with one compromise providing access to perform the next one.

During a White House press briefing in February, Deputy National Security Advisor for Cyber and Emerging Technology Anne Neuberger acknowledged this threat and warned that it will take months for the government to determine the full scope of the attack. “As of today, nine federal agencies and about 100 private sector companies were compromised,” she said. “As you know, roughly 18,000 entities downloaded the malicious update. So, the scale of potential access far exceeded the number of known compromises. Many of the private sector compromises are technology companies, including networks of companies whose products could be used to launch additional intrusions.”

Mike Raeder, an industry veteran who held cybersecurity leadership roles at various organizations and until recently served as deputy CISO at Northrop Grumman, feels that boards need to be better educated about the technical aspects of such attacks. In the long term, he believes boards should diversify themselves by including former CIOs or CISOs.

“Making sure our boards understand the larger picture cyber risk is vitally important and then they will start to ask those larger questions but even if they don’t ask them, it’s the responsibility of cyber leaders to talk to boards about the extent of cyber risk within an entire organization,” he tells CSO. “That just doesn’t mean the IT pillar, but also where cyber risk exists beyond that. When we talk about software supply chains in particular, absolutely we should be asking our software vendors if they use SolarWinds or any [compromised] products. Where’s the risk in your organization that could transfer to my organization?”

2. Does our current security program cover software supply chain threats?

The main problem with defending against software supply chain attacks is that they abuse a trusted relationship between users and vendors and the legitimate access and privileges a particular piece of software is given to perform its function. From the user’s perspective, the software they download came from a reputable source through the correct distribution or update channel and is digitally signed. Users can’t be expected to reverse-engineer and analyze code on the software updates they deploy in their infrastructure and regular companies are not security vendors, so they don’t have employees with those skill sets.

Organizations must assume that they probably won’t be able to detect the initial software supply chain intrusion. However, they can take steps to block and detect the second stages of the attack. This includes attempts to download additional tools and payloads, attempts to communicate to external command-and-control servers and attempts to move laterally to other systems.

3. If the government and security vendors like FireEye were compromised, how can we protect ourselves?

Companies need to focus on the maturity of their security programs, Webb says. Everyone starts by building a strong outer shell with firewalls, intrusion detection and prevention systems, DNS control and other things that create boundaries, but they don’t put a lot of effort into hardening the inside of their environments, he says.

Companies could detect lateral movement activities such as attempts to abuse administrative credentials, but that usually requires advanced monitoring and behavioral detection tools and large security operations centers that are outside of the price range for small- and medium-sized organizations. What these organizations can do, according to Webb, is make sure that they cover the basics and all systems and internal environment is as hardened as possible.

For example, at Avalon, communications on the internal network are encrypted so that in case of a compromise, attackers can’t extract credentials or useful information from the traffic even if they can intercept it. All DNS traffic is forced through a firewall and a DNS filter and all URLs that servers are allowed to connect to need to be whitelisted. This ensures that if a server is compromised the malicious code can’t reach a command-and-control server to download additional tools or commands.

When deployed, all servers go through a hardening process where everything is locked down and blocked and then connections are whitelisted as needed. It’s the same with databases. A server accessing a database is only able to view what it needs to do to perform its job. Updates are only served from an internal server and not directly from the internet, preventing a compromised server from opening external connections. Users never log in with admin credentials and can only use whitelisted applications. End users, Windows servers, and Linux servers are on separate directories so that if one is compromised, the whole environment is not compromised.

“Seven years ago, I started applying zero-trust type principles and it’s how we’re designed,” Webb says. “A zero-trust principle is really managing the device down to only what you trust and only what its function is supposed to be. Everything else is not trusted. Everything else should be blocked. If something tries to violate that, it should be a big red alarm: Why is this system trying to do something I didn’t intend for it to do?”

In fact, FireEye, the company that discovered and reported the SolarWinds attack after its own network was targeted discovered the intrusion because the attackers added a secondary device to one of their employees’ accounts in order to bypass multi-factor authentication. This was flagged as suspicious and the employee was questioned.

“That probably is the answer: zero trust and behavioral management, and if something violates your zero trust, you have to investigate it,” Webb says.

4. Should software supply chain attacks lead to a closer review of vendors and suppliers?

When choosing their software solutions or services some organizations might take into consideration a vendor’s vulnerability management practices: How do they handle external vulnerability reports? How often do they release security updates? What are their security communications like? Do they publish detailed advisories? Do they have a secure software development lifecycle aimed at reducing the number of vulnerabilities in their software? Some companies might also ask for information about penetration tests and other security compliance reports.

However, with attacks like SolarWinds software development organizations need to go beyond that and invest in better securing their own development infrastructure and environment because they are increasingly a target for attackers and can be the victims of software supply chain attacks themselves through the tools and software components they use. They also need to consider the risks posed to users during the design stage of their applications and try to limit the impact a compromise can have by limiting the privileges and access the applications need to perform their function.

“Our responsibility is going to be to apply pressure to the supply chain: You have to step up your game. You’re going to have to put these things in place. You’re going to have to, you know, test and audit your own code, not on an annual basis but probably like a monthly basis or prior to any deployment,” Webb says. “Business leaders need to press on IT leaders to make sure they are only dealing with reputable vendors and people that have taken the next step in protecting the code their organization is using.”

According to Larry Schwarberg, vice-president of information security at the University of Phoenix, a first step in that direction for many organizations will be to identify all their software and SaaS suppliers and have a clearly defined onboarding process for new applications and services. Many organizations have a shadow IT problem where different teams buy and deploy hardware and software assets without those vendors being vetted by the security team. That poses a serious problem for locking down applications and enforcing least privilege access.

As contract and subscription renewals come up, organizations should ask their software and service providers those deeper questions about pentesting and how they are testing their software and limiting potential impact, Schwarberg says.

Running thorough assessments of software suppliers from a supply chain security perspective is not easy and requires resources and expertise many companies might not have, but according to Raeder, audit organizations are likely going to use this event as a catalyst to build more capability around this.

“I think a lot of organizations today might do a cyber questionnaire and who knows what’s done with those results when they get them,” Raeder says. “They may be reviewed or they may go into a scoring system. On both sides of the equation here, this is an opportunity to get better at supply chain risk analysis. Organizations need to look at their supply chains and figure out who are their most critical suppliers that they need to spend the most time with analyzing risk and suppliers need to be able and prepared to answer those cyber questions in an effective way for their customers.”

5. Is this type of attack only used by APT groups and nation-states? 

Many of the high-profile software supply chain attacks to date have been attributed to APT groups with suspected ties to governments, including the SolarWinds attack and the ShadowPad attack. According to an analysis by the Atlantic Council of 115 software supply chain attacks and vulnerabilities disclosed over the past 10 years, at least 27 of them were state sponsored.

However, the skills and resources required to pull off software supply chain attacks are not necessarily only limited to traditional cyberespionage groups. Various attacks over the years have involved backdoored open-source components or backdoored versions of legitimate applications served from compromised download servers that were financially motivated and were likely launched by cybercriminals. There was even a case tied to ransomware.

Over the past few years, many ransomware gangs have adopted sophisticated techniques that in the past were only seen with APT groups, including in-memory process injection, deep reconnaissance, manual hacking and lateral movement using system administration tools, fileless malware. Some ransomware groups have also targeted managed service providers (MSPs) in attacks that are similar in concept to software supply chain attacks: targeting organizations that can provide privileged access into other companies due to their business relationship.

An increasing number of cyber mercenary groups sell hacking services to both governments and private entities in the cybercriminal underground. As more groups start adopting this attack vector, all organizations, big or small, regardless of the industry they operate in, could become the target of an APT-style attack through a software supply chain compromise.