Millions of devices, from consumer products like printers and IP cameras to specialized devices used across organizations such as video conferencing systems and industrial control systems, are at risk due to critical vulnerabilities found in an embedded TCP\/IP library. Some of the flaws allow for remote code execution over the network and can lead to a full compromise of the affected device.The vulnerabilities were found by an Israeli company called JSOF that specializes in the security of IoT and embedded devices. They affect a proprietary implementation of network protocols developed by a company called Treck. The researchers found 19 flaws, several of which are rated critical, and have dubbed them Ripple20 because they were reported in 2020 and have a ripple effect across the embedded supply chain.JSOF worked with researchers from IoT security and visibility firm Forescout to identify potentially affected products by using TCP\/IP network signatures in its large knowledgebase of embedded devices. The researchers also worked with ICS-CERT, the critical infrastructure arm of the US Cybersecurity and Infrastructure Security Agency (CISA), to notify and confirm affected products and vendors.So far, products from 11 vendors have been confirmed as vulnerable, including infusion pumps, printers, UPS systems, networking equipment, point-of-sale devices, IP cameras, video conferencing systems, building automation devices, and ICS devices, but the researchers believe the flaws could impact millions of devices from over 100 vendors.Memory corruption vulnerabilitiesAll the vulnerabilities are memory corruption issues that stem from errors in the handling of packets sent over the network using different protocols, including IPv4, ICMPv4, IPv6, IPv6OverIPv4, TCP, UDP, ARP, DHCP, DNS or the Ethernet Link Layer. Two vulnerabilities are rated 10 in the Common Vulnerabilities Scoring System (CVSS), which is the highest possible severity score. One can result in remote code execution and one in an out-of-bounds write. Two other flaws are rated above 9, meaning they're also critical and can result in remote code execution or the exposure of sensitive information.Even if rated lower, the remaining vulnerabilities might be serious, as CVSS scores don't always reflect the risk to actual deployments based on the type of devices. For example, in a critical infrastructure or healthcare setting, a denial-of-service vulnerability that prevents a device from performing its vital function can be seen as critical and could have disastrous consequences.When it comes to critical infrastructure the CIA triad of security properties\u2014confidentiality, integrity and availability--is reversed and you worry about availability more because operations need to be running, for example, at a railway, at a gas pipeline or in a manufacturing plant, Daniel dos Santos, a research manager at Forescout, tells CSO."The reason why a denial-of-service issue would still not be considered critical in critical infrastructure is that there are simply too many of them," Shlomi Oberman, the CEO of JSOF, tells CSO. "There are all sorts of resource consumption issues that are not being solved, and we have a long way to go and a big fight to fight until we get there. We're still trying to reach a state where everybody at least fixes their remote code executions."Supply chain complexityThe Ripple20 flaws highlight the difficulty of understanding the scope of security vulnerabilities in the IoT and embedded device world due to the complex supply chain and a lack of a software bill of materials in the development process. Some affected vendors were not even aware they had this TCP\/IP library in their products, because it was actually used by a third-party hardware module or component that was part of their devices.An example of that are medical devices from Baxter, which are vulnerable because they use hardware modules from Digi International, a large system-on-module (SoM) manufacturer, which uses the Treck library in its components.Most operating systems have their own networking stacks, but this is not always true in the embedded world, where a hardware component might not run a full operating system yet could still have network connectivity built in.Treck is one of a few independent developers of low-level network protocols for embedded devices with implementation of ICMPv6, IPv6, TCP, UDP, ICMPv4, IPv4, ARP, Ethernet, DHCP, DNS and more. Its TCP\/IP stack has been around for around 20 years and the complex supply chain relationships have created a fragmentation problem. Different versions of the library ended up in a variety of products, some directly, some indirectly through a component supplier.Some suppliers might have long gone out of business, were acquired by other companies, or ended their production of those components. Some of the affected products might have reached end-of-support or are hard to patch because they don't have easy update mechanisms. Others might be serving critical functions in factories and industrial installations and can't easily be taken offline to be updated.The JSOF researchers said a few of the issues they found exist only in older versions of the Treck TCP\/IP stack and have disappeared over the years due to code rewrites. However, those code changes were not necessarily intentional security fixes, so customers did not treat them as security updates. Vulnerable and old versions of the library are still used by devices in the wild.Most of the vulnerabilities, though, including the critical ones, were zero-days when they were discovered, meaning they affected even the latest version of the library, so affected vendors should update their products, which is not always an easy process. Treck has developed patches for all the vulnerabilities, but not all affected vendors had support contracts with the company, so they had to renew their contracts, Shlomi Oberman tells CSO.Total number of affected vendors unknownThat's only for the big vendors who were able to confirm they are affected. Many others were not even able to confirm that they are affected. Just like with the URGENT\/11 vulnerabilities that were disclosed last year in the IPnet TCP\/IP stack of VxWorks, a widely used embedded real-time operating system (RTOS), Oberman expects more vendors will confirm that they are vulnerable to Ripple20 as time goes on. "There is a list of around 100 potentially affected vendors and only about 15 have confirmed so far," he says. "We estimate that hundreds of millions of devices are affected."The confirmed vendors include HP, which uses the library in some of its printers; Hewlett Packard Enterprise (HPE); Intel, which uses the stack in the AMT out-of-band management firmware for Intel vPro-enabled systems; Schneider Electric, which uses Treck in its uninterruptible power supply (UPS) devices and potentially other products; Rockwell Automation; medical device manufacturers Baxter and B. Braun; construction and mining equipment manufacturer Caterpillar; IT services firm HCL Technologies; and component manufacturer Digi International.To exacerbate the supply chain problems, a separate variant of the Treck TCP\/IP stack called KASAGO is commercialized in the Asian marker by a company called Elmic. That, too, likely has many of the same vulnerabilities and adds to the supply chain complexity.JSOF and Forescout have worked to develop signatures based on traffic patterns that could be used to identify potentially vulnerable devices. On top of that, they did a lot of open-source intelligence gathering by analyzing legal and copyright documentation for products, looking for mentions of Treck in stack traces and debugging symbols during firmware analysis or discovered business relationships between the library developer and various vendors on LinkedIn.Forescout added the detection capability to its own IoT visibility and management products and JSOF plans to release some of the information so that businesses can develop scanning and monitoring capabilities for their own networks to identify devices that might contain the affected Treck library and isolate them. Devices that are exposed directly to the internet are at immediate risk, but these vulnerabilities can also be exploited for lateral movement through networks and the compromised devices could serve as a persistent foothold for attackers.A Shodan search for 37 affected device models from 18 vendors performed by Forescout, revealed around 15,000 devices that are directly connected to the internet and could be potentially compromised by anyone.The idea of having a software bill of materials for all technology products similar to the labels on food products could be interesting, but so far it doesn't exist in the real world, dos Santos says. "So, what you can do is to have a sort of network monitoring approach like the one we did at scale, but for your organization to see in your group, what devices you have and what devices are potentially impacted by using traffic patterns and signatures and so on like the ones that JSOF developed."The JSOF report contains additional mitigation advice.