If there\u2019s a poster child for the challenges facing open source security, it may be Werner Koch, the German developer who wrote and for the last 18 years has toiled to maintain Gnu Privacy Guard (GnuPG), a pillar of the open source software ecosystem.Since its first production release in 1999, GnuPG has become one of the most widely used open source security tools in the world, protecting the email communication of everyone from government officials to Edward Snowden.Yet Koch found himself struggling to make ends meet in recent years. The estimated $25,000 he collected on average in annual donations since 2001 weren\u2019t enough to support his efforts. As reported by Pro Publica, the 53-year-old was close to throwing in the towel on GnuPG when Edward Snowden\u2019s NSA revelations shocked the world, convincing Koch to soldier on. "I'm too idealistic," he said.The story has a happy ending. After the ProPublica story broke, donors from around the world rushed to support Koch. He easily surpassed the $137,000 fundraising goal he had set to support his work, enabling him to hire a part-time developer. Koch was awarded a one-time grant of $60,000 from the Linux Foundation\u2019s Core Infrastructure Initiative. Facebook and the online payment processor Stripe\u00a0each pledged $50,000 a year to Koch\u2019s project.Underfunded projects, as GnuPG was until recently, form part of a vast open source ecosystem unprecedented in scale. Widespread reuse of open source code fuels today\u2019s surging technology development, but the sheer volume of that code discourages security vetting. Only recently have we begun to confront the problem, often on the heels of security breaches that embarrass the industry into action.Will code for foodThe conditions that left Koch high and dry for years are not unusual.After Google researcher Neel Mehta uncovered Heartbleed, a serious and remotely exploitable vulnerability in a component of OpenSSL, the software community was shocked to learn that the project was largely the responsibility of what Jim Zemlin, executive director of the Linux Foundation, referred to as \u201ctwo guys named Steve.\u201d Dr. Stephen Henson and Steve Marquess labored part-time to keep the code up to date, compensated by a few thousand dollars a year in voluntary contributions.Technology vendors who rely on open source were quick to swoop in and set the OpenSSL project to rights. The Core Infrastructure Initiative that gave GnuPG\u2019s creator a $60,000 grant was established months earlier to help fund the work of Henson and others on OpenSSL. Financial support is provided by such Silicon Valley giants as Amazon, Adobe, Cisco, Facebook, and Google.A thousand eyes blinkHeartbleed wasn\u2019t the first killer open source bug. For example, the Apache Struts vulnerability predated it by almost a year and was at least as serious.But thanks to the media frenzy, Heartbleed may have permanently discredited Eric Raymond\u2019s famous adage about open source quality: \u201cGiven enough eyeballs, all bugs are shallow.\u201d Most security professionals say the notion was always more aspirational than descriptive.\u201cI never liked the \u2018with many eyeballs\u2019 notion,\u201d says Joshua Corman, CTO at the firm Sonatype. \u201cJust saying there are \u2018many eyeballs\u2019 doesn\u2019t mean that those eyeballs are motivated or qualified to look to find security vulnerabilities.\u201dOpen source\u2019s \u201cmany eyes\u201d assurance served mostly to gloss over a weakness in the open source ecosystem, implying an atmosphere of constant vigilance where none existed, says Bill Weinberg, senior director of open source strategy at the firm Black Duck Software.\u201cWith Shellshock, you didn\u2019t have many eyes,\u201d Weinberg says, referring to critical vulnerability that was found lurking in code for Bash in 2014. \u201cThat code was considered to be well vetted, but it turned out that it wasn\u2019t well curated, because everyone assumed it was well vetted.\u201dWhile we might like to assume that the integrity of open source code is high, data from Sonatype suggests the opposite. An analysis by the company of open source components in its managed codebase found that known vulnerabilities in open source components were remediated only 41 percent of the time, Corman wrote in the Usenix journal ;login. For the issues that were fixed, the mean time to remediate them was a whopping 390 days.But even talking about \u201copen source\u201d apart from commercial, proprietary software is misleading. While a line may have once separated open source and proprietary software projects, most modern applications are assemblages of third-party software components, many of them open source, Corman said.Getting serious about code-level securityWhat\u2019s the proper response? For better or worse, the answer is largely cultural, says Katie Moussouris, chief policy officer at the firm HackerOne and a former senior security strategist at Microsoft. \u201cWe need to build a security mind-set. This is important to every software project -- open source or not.\u201dMoussouris\u2019 company provides a Web-based platform for coordinating vulnerability disclosures, including through bug bounty programs. She notes that HackerOne already sponsors bug bounties for a wide range of open source projects, including PHP, Ruby on Rails, Python, and OpenSSL, providing compensation for reports of vulnerabilities.Open source projects need to take a more serious and systematic approach to security, she says: \u201cYou have to at least try to build security in.\u201dSonatype\u2019s Corman advocates a more rigorous solution: a supply chain akin to those used by manufacturers that delivers both high quality and accountability.Using the analogy of a Ford manufacturing line, Corman notes that the company knows the provenance of each part that goes into its finished automobiles. Problems can be traced back to specific suppliers, facilities, and even production runs.In modern software development organizations, however, no such system exists. Almost every commercial software application makes use of open source and proprietary components sold by third-party firms, but software firms may have a cursory notion of the quality and origin of all that code. Often, the reach and impact of vulnerabilities become known only after disaster strikes.In the case of Shellshock, for example, the code in question dated back to 1989 and impacted a wide range of applications -- from CGI-based Web servers to Qmail email servers to certain DHCP clients. Attacks against the vulnerability began within hours of its disclosure.\tFollowing the leadersOrphan projects and lax inspection should not obscure that some leading lights in the industry are well along the virtuous path to secure code.Commercial Linux firms like Canonical, Red Hat, and Google already invest heavily in the security and integrity of open source. Wealthy, open-source-friendly firms such as Netflix and Facebook have directed considerable resources to projects that improve open source quality as well.At Mozilla, responsibility for security is split among three teams, according to Engineering Manager Jason Duell. One team triages security issues as they're discovered. A second does \u201cfuzzing\u201d (black box testing of compiled code) to find vulnerabilities, and a third team develops such security and privacy features as Mozilla\u2019s Content Security Policy.Duell says that fuzzing and rigorous testing of developed code has been a mainstay of Mozilla\u2019s development culture since before he arrived six years ago. But Mozilla has changed other development practices as awareness of the threat to open source has grown.\u201cWe've changed various practices to be inline with the fact that adversaries are watching our public source repository for commits,\u201d Duell says. For one, security fixes to Mozilla code are coordinated to land prior to release to give attackers a smaller window to work with. Large new features get security team audits prior to release.At Canonical, which makes Ubuntu Linux, a fast-growing security team audits Canonical\u2019s code -- a total of 35,000 software packages that are released as part of Ubuntu through a variety of channels, according to Dustin Kirkland, Canonical\u2019s Cloud Solutions Product Manager.As with Mozilla, Canonical\u2019s security operations span a number of different initiatives, from feature development to code audits. To Corman\u2019s point, Kirkland says that supply chain risk is a major concern. Canonical devotes considerable resources to assessing the viability of open source components that are bundled with its core operating system.\u201cWith open source technology, we\u2019re looking at whether it's better to adopt it or to fork it, then develop and extend it,\u201d said Kirkland, who is a 20-year open source veteran and distribution maintainer of more than 20 open source projects. The company has come under fire from within the open source community when it opts to fork existing open source code, but Kirkland cites the ability to fork as one of open source\u2019s strengths.\u201cWe\u2019re not going to create own versions of OpenSSL and GPG,\u201d says Kirkland. \u201cBut having alternatives to encryption libraries is important. There needs to be diversity, especially as we\u2019re learning how vulnerable some of these components are.\u201dWe\u2019re all open source nowAs uncomfortable as that may be, experts say there\u2019s no going back. Weinberg spent a good part of his career as what he calls a \u201cdefender of the faith,\u201d countering attempts by commercial vendors like Microsoft to discredit the open source movement. He says the wall that once separated \u201copen source\u201d and \u201cclosed source\u201d was torn down long ago.\u201cThere\u2019s no such thing as proprietary software anymore because there\u2019s very little software without some dependency on open source,\u201d he said. \u201cThe world has moved to \u2018community developed\u2019 software in one form or another.\u201d\u201cI really think it\u2019s a shared responsibility,\u201d says Kirkland of Canonical. \u201cWhen you consider how dependent all of us are on a whole stack of open source software, you have to hope that [security] becomes a shared responsibility and that it isn\u2019t left up to the the Linux Foundation and Red Hat to figure out this stuff.\u201dIn other words, we can gnash our teeth and tear at our hair over the likes of Heartbleed, but in 2015, all companies that make, use, or rely upon software are de-facto open source software companies whether they know it or not. That makes them part of the problem and its solution.