Recent cyberattacks leveraging weaknesses in continuous integration\/continuous delivery (CI\/CD) pipelines and developer tooling warrant a need for increased security of the developer infrastructure. Prominently, the Codecov supply-chain attack has alerted everyone against storing secrets in CI\/CD environment variables, no matter how safe the environment might be.By breaching a Bash uploader used by thousands of developers, Codecov attackers managed to siphon credentials, keys, and API tokens from customer environments, remained undetected for two months, and further, reportedly breached hundreds of restricted customer networks. Likewise, attacks on automation tools like Jenkins, GitHub Actions and cloud-native containerized environments have further prompted companies to explore and deploy effective defenses for these tools.Below are some best practices to ensure your CI\/CD pipelines remain secure.1. Stop storing secrets in CI\/CD environmentsThe reason behind the large success of Codecov supply-chain attack remains that the environment variables exfiltrated by the attackers contained hardcoded secrets including passwords, tokens, and keys. Because some of these credentials gave attackers access to the companies\u2019 private GitHub repositories, further data exfiltration could occur from these private repos that contained data that should have remained confidential.\u00a0Although multiple Codecov customers including HashiCorp, Twilio, Rapid7 and Monday.com disclosed the impact from the supply-chain attack, the most significant data breach to come to light thus far occurred at Japanese e-commerce giant Mercari.\u00a0More than 27,000 total records pertaining to Mercari customers\u2019 finances, merchants, business partners, company employees, contractors, and various entities were exposed to unauthorized external actors following the Codecov attack.Granted, each of these attacks may have started from the Codecov breach, some have questioned why personally identifiable information (PII) such as customer financial records were stored in private GitHub repositories.Similar concerns have been raised with HashiCorp\u2019s GPG private key stored in the CI\/CD environment. This is the secret key that was used to sign and verify software releases published by HashiCorp. Before the key was revoked, an attacker could have abused the key to forge HashiCorp\u2019s signature on a malicious software release.\u00a0\u201cWhy is no one talking about the fact that HashiCorp, the maker of Vault, had their signing key stored as an ENV? Dear me\u2026 This makes me feel better about my life,\u201d tweeted a developer.Organizations need to rethink what secrets can be stored in CI\/CD tooling, environment variables, and private GitHub repositories. If an application requires a credential or token to be stored in these places, it\u2019s best to store credentials to an account or resource with the lowest privileges, just what\u2019s necessary to accomplish the task\u2014often referred to as the principle of least privilege. That way, even if the secret does get exposed in an unprecedented attack, the damage is contained.2. Scrutinize automated pull requests and scheduled tasksCI\/CD automation tooling like GitHub Actions allows developers to set up scheduled tasks for their code repositories, such as automatically vetting and processing incoming pull requests. But what happens if the contributor making the pull request to an open-source project has malicious intent?In April 2021, GitHub Actions was abused by attackers who made automated pull requests to hundreds of repositories with the intention of mining cryptocurrency using GitHub\u2019s infrastructure. This large-scale attack occurred after the \u201cflaw\u201d with GitHub Actions had been reported earlier in February.At a minimum, these pull requests can abuse GitHub\u2019s servers to mine cryptocurrency or execute the attacker\u2019s malicious code. If the project owners are negligent and merge these pull requests, they have now introduced malicious code into their repository and the wider software supply chain. In May, GitLab reported tackling similar cryptomining attacks on its platform from attackers abusing \u201cfree minutes\u201d (quota) allotted to new accounts.Because the very nature of CI\/CD automation tools like GitHub Actions and GitLab is to provide ease of automating key tasks, gatekeeping becomes a challenge. What may have been an intentional feature soon turns into a security flaw, following abuse by threat actors.GitHub recently announced new features to combat abuse of its Actions platform by cryptomining attackers: \u201cPull requests from first-time contributors will require manual approval from a repository collaborator with write access before any Actions workflows run. When a first-time contributor opens a pull request, they\u2019ll see a message that a maintainer must approve their Actions workflow before it will run,\u201d stated GitHub product manager Chris Patterson in a blog post.Leading CI\/CD solutions and DevOps platforms can follow GitHub\u2019s lead in adding some security checks in place to deter large scale abuse of their infrastructure by malicious actors.3. Harden and periodically audit your cloud-native containersNothing beats following standard best practices such as ensuring your production containers are properly configured and hardened against common attack vectors. This includes securing your pipeline configuration.However, simple misconfiguration errors sometimes become hard to spot by humans. Then comes the question, are your Docker-based environments free from vulnerabilities? This is why regularly performing a security audit of your containers for weaknesses, scanning container images and manifest files to spot common security issues remains helpful.Investing in dependable cloud-native container security solutions that can automate much of this is advisable. The huge volume of security vulnerabilities that are reported every year makes it virtually impossible for a human to keep a track of them.Additionally, as companies adopt Kubernetes frameworks and Docker containers to deploy their applications, container security solutions with built-in web application firewalls can detect and block suspicious network traffic early on. This can help prevent a bigger compromise, even if the attackers are able to penetrate a container and gain initial access.4. Integrate deep code scanning to automate code quality checksHaving tooling in place to automatically spot code quality issues, security vulnerabilities, and bugs like memory leaks or race conditions before code commits reach production is an effective strategy in securing your CI\/CD pipeline from the start. Although the focus seems to be primarily on preventing cyberattacks, innocuous bugs are just as likely to have large-scale impact. We recently saw this with the global Fastly outage that knocked major sites around the world offline.Solutions like GitHub code scanner or Sonatype\u2019s Lift [full disclosure: Sonatype is my employer] integrate seamlessly within your existing coding workflow and provide basic safeguards at no cost to developers. Ultimately, an organization\u2019s goal should be to support its developers to do their best work, while preventing the introduction of bugs or security vulnerabilities in the applications as much as practically possible. This needs to happen while causing minimal friction between the development and security teams. This is where real-time notifications alerting the developer of a potential oversight as they code can save everyone time and secure the overall CI\/CD workflow from the beginning.\u00a05. Patch early against latest CI\/CD tooling vulnerabilitiesIn March 2021, attackers employed a cryptomining botnet called z0Miner to mine Monero (XMR) cryptocurrency on vulnerable Jenkins and ElasticSearch servers. By exploiting remote code execution (RCE) vulnerabilities in internet-facing servers, attackers attempted to infect and take over the automation infrastructure to conduct their nefarious activities.Likewise, as reported last year, Jenkins servers could be exploited by attackers to rapidly cause distributed denial of service (DDoS). This resulted from a UDP amplification reflection DoS vulnerability, tracked as CVE-2020-2100, which affects versions below v2.219 of Jenkins, and below 2.204.1 of Jenkins LTS.Patching automation tooling and pipelines against these severe vulnerabilities as soon as these are discovered remains vital to ensuring the security of your CI\/CD infrastructure.\u00a06. Verify integrity of updates before applying themApplying latest updates and patches is sound advice, but are you sure the update you are receiving has not been tampered with? The advice to just relentlessly \u201cupdate to the latest version\u201d which has been the mantra of security professionals for decades has been challenged following the SolarWinds supply-chain attack.In the SolarWinds\u00a0incident, malicious updates made to the Orion IT product enabled attackers to distribute malicious code downstream to over 18,000 customers. Yet again, Passwordstate password manager\u2019s \u201cIn-Place Upgrade functionality\u201d had been compromised to distribute malicious updates to Passwordstate users. Therefore, blindly applying product updates is a bad idea.In Codecov\u2019s case, a simple integrity check led to the discovery of the two-month long breach. A customer noticed a discrepancy between the checksum (hash) of the Bash Uploader that was hosted on the server and the legitimate checksum listed on Codecov\u2019s GitHub repository immediately contacted Codecov which remedied the problem.\u00a0As such, a defense in depth approach warrants that the integrity of any updates, patches, and downloads be verified so that risks from a sophisticated supply-chain attack can be ruled out.