The compromise of the attacker on the Apache Software Foundation’s (ASF) servers should be used as a training vehicle for all security and server teams. The reason it'simportant is not that ASF was compromised. Rather it’s because ASF was open about what happened and what they intend to do about it.
The attack process was described in detail in a recent article.
The hack started with the compromise of apachecon.com, a website that's owned by the ApacheCon conference production company. Although logs confirming the exact cause were destroyed, investigators suspect it was the exploit of one or more local root vulnerabilities in the Linux kernel for which Red Hat issued a patch seven days earlier but had not yet been installed. They then used the SSH key for a backup account to access the server that runs people.apache.org.
With an unprivileged user account, the attackers added common gateway interface scripts to the document root folders for several Apache websites. Routine backup processes then copied the scripts to the foundation's production server, where they became visible to the outside world. Those scripts, which allowed the hackers to obtain remote shells, were aided by Apache's use of ExecCGI.
Source: Breaching Fort Apache.org – What went wrong?, Dan Goodin, The Register, 3 Sep 2009
Attackers used a known vulnerability to hack the initial server, which didn’t even belong to ASF. It was owned by ApacheCon, a conference production company. Should the Red Hat patch have been applied? Maybe. However, it isn’t unusual for an organization to fully test a patch before deploying it. A 30-day window is not uncommon. So I don’t believe negligence is to blame.
The takeaway from this is more about the 24/7 vulnerability of all networks and the questions we must ask whenever we deploy technology. We simply have to assume someone will find a way to take control of one of our servers, using it as a platform from which to take control of some or all of the rest of our systems.
The real issue in this breach was the way in which SSH keys were managed, as written about in ASF’s postmortem.
- The use of SSH keys facilitated this attack. In hindsight, our implementation left a lot to be desired--we did not restrict SSH keys appropriately, and we were unaware of their misuse.
- The rsync setup, which uses people.apache.org to manage the deployment of our websites, enabled the attackers to get their files onto the US mirror, undetected.
- The ability to run CGI scripts in any virtual host, when most of our websites do not need this functionality, made us unnecessarily vulnerable to an attack of this nature.
- The lack of logs from the ApacheCon host prevents us from conclusively determining the full course of action taken by the attacker. All but one log file were deleted by the attacker, and logs were not kept off the machine.
ASF is correcting these gaps. But the lesson for all of us is to keep an eye on how our teams configure each system or service, whether with Linux or some other environment. Managers asking the right questions, even when everyone claims the network is secure, is the first step. Regular vulnerability scans and penetration tests by disinterested parties is the next.
Would the attack have been successful if none of these conditions had been present? Maybe. But it would have taken another form and potentially presented a level of difficulty too high to bother.
Finally, I want to thank Apache for its openness and willingness to share what happened with the rest of us.