Facebook bug hunter stumbles on backdoor left by… another bug hunter

When Orange Tsai set out to participate in Facebook’s bug bounty program in February, he successfully managed to gain access to one of Facebook’s corporate servers. But once in, he realized other hackers had beaten him to it.

Tsai thought he had stumbled on some malicious activity in Facebook’s network. But, according to a statement from Facebook on Friday, what he found was something else.

Tsai, a consultant with Taiwanese penetration testing outfit Devcore, had started by mapping Facebook’s online properties, which extend beyond user-facing services like facebook.com or instagram.com.

One server that caught his attention was files.fb.com, which hosted a secure file transfer application made by enterprise software vendor Accellion and was presumably used by Facebook employees for file sharing and collaboration.

Tsai analyzed the application and found seven vulnerabilities, including two remote code execution ones, which he reported to Accellion.

He used the vulnerabilities to gain access to Facebook’s corporate server and started to gather information from its logs in order to prepare a report for the company’s security team.

That’s when he spotted some unusual errors in the server’s log that pointed him to what turned out to be a PHP-based backdoor — also known as a Web shell — that had previously been installed on the server.

The rogue Web shell allowed those who installed it to execute shell commands on the server and to upload files, but more importantly, it hijacked the Accellion application’s authentication process and recorded the credentials of Facebook employees accessing it.

“At the time I discovered these, there were around 300 logged credentials dated between February 1st to 7th, mostly ‘@fb.com’ and ‘@facebook.com’,” Tsai said in a blog post Thursday. “Upon seeing it I thought it’s a pretty serious security incident.”

Since the Accellion application allowed authentication through LDAP and Windows Active Directory, Tsai suspected that the captured Facebook employee credentials might also work for other Facebook corporate servers, such as Outlook Web App or VPN, but he didn’t try to use them.

Further analysis of the server logs revealed that the hackers who originally installed the Web Shell downloaded the captured credentials and deleted the file containing them every few days. In addition, there was evidence that they tried to map Facebook’s internal network, to log into the LDAP and other servers and even searched for SSL private keys.

However, Facebook had a different explanation for what Tsai discovered on its servers. In a statement, it said the “hackers” Tsai had detected was actually another researcher from Facebook’s bug bounty program doing their own testing.

It said the vulnerable application Tsai discovered was not connected to its core network or its production systems. And it said the employee credentials on their own “don’t provide access to critical systems, which are protected through 2-factor authentication.”

Still, Tsai was correct that someone else had been inside Facebook’s network before him.

“There were two periods that the system was obviously operated by the hacker, one in the beginning of July and one in mid-September,” he said.

The July incident happened to take place right around the same time that a remote code execution vulnerability in the Accellion File Transfer Appliance was publicly disclosed.

Tsai reported all of his findings to Facebook, which awarded him a $10,000 bug bounty and launched its own forensics investigation that was completed this month, prompting the researcher’s blog post.

This story has been corrected to reflect new information that was subsequently provided by Facebook.