In part one of our series on the collision of search engine optimization and black-hat hacking (see "Black Hat SEOs: Is This the Future of Search?"), we explored how search engine optimizers, or SEOs, have learned tricks that change the search results that drive much of the traffic to successful websites. (The practice of search engine optimization is also called SEO.) Many of these upstart entrepreneurs have made small fortunes as SEO consultants. Many also use SEO to drive traffic to their own sites that sell products, ads or referrals--a business known as search marketing.
We explored how the tactics of SEO include some unsavory ones that range from digital ffibs to aggressive deception. The tricks are called black-hat SEO, though that's something of a misnomer since, as SEOs like to say, they don't break the law, just the search companies' terms of service. The search companies tried to stay ahead of black-hat SEO by tweaking their algorithms and adding filters that penalize sites for questionable tactics. Increasingly, though, it looked as if the combined forces of SEO and black-hat hacking would be too much for any algorithm....
As search companies have tried to contain the more aggressive techniques that SEOs were using to manipulate search-engine rankings, black-hat SEOs have responded by circumventing the rules. Rather than just using loopholes, they began actively abusing the algorithms used to determine search engine results. The tactics became so aggressive that the SEOs started to make the search engines look bad: Search results started to reflect the SEO's reality, rather than a reality that rewarded good sites. Like all arms races, this one eventually escalated to an untenable level. The game had to change again. And it did, about 18 months ago.
Suddenly and without much warning, search companies--Google especially, the SEOs say--decided to enforce its terms of service, and severely. The algorithms wised up some, but more than that, it appeared that Google was buttressing its algorithm with filters and manual labor. If enough complaints came in about a site using black-hat tactics, Google would manually adjust the rankings or simply blacklist the site--a process SEOs call a "hand job."
Some SEOs and search marketers were surprised. The top SEOs generally maintained good lines of communication with Google and other search companies. Some, like Jeremy Schoemaker--a search marketer known online as Shoemoney--would even periodically ask for advice on SEO techniques and whether they'd get him in trouble.
But now the search companies were matching the SEOs' aggressiveness. The effect could be devastating. A site that was blacklisted lost its traffic, and therefore its business, overnight. Usually targeted sites clearly violated search terms of service. But some weren't doing anything differently than they'd been doing for months or years. "When people are ranking for a phrase and supporting their family, and then the next day they're off the map, that's really vicious," says Schoemaker. "You can literally ruin someone's life."
Of course, Google could make the argument that turnabout is fair play. Perhaps enforcement was brusque and arbitrary, but so is black-hat SEO. Nothing Google was doing was illegal, which was an argument the black-hat SEOs had made for years. Plus, as early as 2006, Matt Cutts, Google's chief liaison to the SEO community, had blogged about the ramp-up in enforcement against overly aggressive SEO.
Even before that, the veteran SEO Eric Ward warned others that eventually the free ride would end. Ward was notorious for his cautious, by-the-book approach to link-building strategies. Some called him "a poser," "arrogant" and "retarded," and bestowed him with the nickname "Linkmoses."
"I understand why [the search engines] are doing it, but their enforcement has become a little heavy-handed," says SEO Michael Gray. Says Aaron Wall, another SEO: "Google went on a crusade."
The Aftermath of the Crusade
As frustrating as delisting was for companies suddenly punished by SEO enforcement, getting relisted proved to be a much worse problem. SEOs and site owners found themselves stuck with little communication from the search companies about what they had done wrong or how to fix it to get back in the good graces of the algorithms. Schoemaker himself lost the top spot for ring-tone searches.
"I was making thousands of dollars a day, and then one day I was out of Google," he says. "I inquired why and never really got an answer. They said it was normal search engine fluctuation"--fluctuation, he notes, that also can be caused by black-hat SEOs. "I probably got gamed out," he suggests. He currently ranks about tenth in ring tones.
Google also partnered with Stopbadware.org to blacklist sites that were potentially infected with malware. Last September, a Web-hosting company in Thailand was hacked and several sites that used the host were flagged on Google, so that if users clicked on a link to the site, an intermediate screen popped up warning them that the site they were about to visit was potentially infected.
Obviously, people rarely visit a site after that kind of warning. The owner of the hosting company, Daniel Peterson, says that after he had cleaned up the sites, nothing had been done to get those blocked sites relisted in Google search results. "No one seems to want to do anything, and the blacklisting is now seriously damaging our businesses," Peterson wrote in an e-mail.
He is particularly concerned about a boutique hotel in Pattaya called Rabbit Resort. Peterson wrote: "Rabbit Resort seriously relies on their Google listing and normally receives 50 to 60 visitors every day. Most of these become bookings. They now receive one every day or so. With more than 60 staff to employ, they now risk financial ruin and disaster." (The sites were eventually relisted).
Roger Thompson, the blogger for Exploit Prevention Labs, cites another recent case, in which search results for "saints football club" brought up a number of Australian soccer team sites that were labeled as potentially containing malware. Thompson notes that another site had this happen. "K1-usa.net...used to be the number-one organic result when people searched for k1. They were hacked for about 10 days, and then cleaned, but in the meantime, they had earned the 'This site may harm your computer' label, and over the next 12 months, before the label was removed, their rating slipped, and slipped, until finally it was nowhere on the first three pages."
Most of the soccer sites were marked clean within days, not months, suggesting Google has improved in the relisting game. "We can always try to do better," says Cutts, the Google liaison. "We're trying to be as responsive as possible."
But Thompson notes, "This happens quite a bit, and I must admit that I'm surprised no one has accused Google of damaging their brand."
"Our webmaster guidelines are clear," says Cutts, who noted that Google made this policy in anticipation of problems with sites using others to goose their rankings. "We say that ultimately you are responsible for what's on your site. If the scam is on your page, that's what is causing damage. We'll do whatever we can to try to help, but ultimately if there's spam content on your pages, we're willing to remove that content, and then hopefully cycle that back in when it's cleaned up."
RSnake, a security expert with experience in Web advertising and SEO who runs ha.ckers.org, says that no matter how blunt and overzealous enforcement has become, that's not the problem with Google's approach to enforcement. It's that the policy that Cutts is referring to is ultimately faulty, because it's based on a false premise.
"Google can shut you down at any time," says RSnake. "But there are all kinds of weird things that could happen to you, upstream problems, a proxy goes bad, someone takes over your site, and there's no way for you to explain that it might not be your fault. They're making false assumptions about how the Internet works, which is that the owner of the IP address is always in control of what happens through that IP address." (Indeed, some black-hat SEOs seized on the opportunity and complained about competitors' sites in hopes that they could get them manually pushed out of the rankings.)
Still, Google's policy of flagging sites and aggressively delisting any site using black-hat SEO remains in place, and by January of this year, Ward felt vindicated for his conservative approach to SEO. About the crackdown on black-hat SEO, a gloating Linkmoses (he has embraced the nickname) wrote a blog entry, "Don't Blame Google for Your Linking Failures":
In 2007, many long-practiced link building tactics stopped being effective. Many link building companies and consultants sold the exact tactics/services that are now useless. Why didn't you see this coming, and if you did, why did you sell those services in the first place and what services will you sell now?... Are you really going to tell me you are shocked that Google no longer thinks a link from link-o-matic, link-to-my-loo, and LinksForNoGoodReason.com are of any value? Please. But if you knew that such links would someday lose value, why did you take money for that very service? And if you didn't honestly know such links were pointless, how can you call yourself a link builder? Google's focus on trusted sources is your worst nightmare.
The Devil They Didn't Know
Certainly gray techniques are still being used by SEOs, and they always will be; Schoemaker recently uncovered a ring-tone business that had come up with a way to take up all the Google AdWords paid-links results for any given search. He estimated that the scheme could net $1 million in four months, and he was surprised Google hadn't banned the company yet.
Still, the crackdown has had an effect. It appears to be cleaving the business. Many SEOs are going more white hat, if you will, and a few have decided to go full-out black hat--a phenomenon that security researcher Jeremiah Grossman calls "SEOwN3d!!1", a mash-up of SEO and hacker slang for compromising a site.
Some decided that the free ride was over, and they cleaned up their act. They've adjusted to the new rules of the playground. The noted SEO David Naylor gave up black-hat SEO and even abandoned jobs for which his revenue would be based on traffic volume. Instead he works on retainer and consults for flat fees--trading in the potential for periodic obscene windfalls for a less outrageous, more stable income. "If I slip off that first page, I still get paid now," he says. "And I've got a team of guys I've got to feed. It was a total business decision."
Cutts of Google believes this is the primary trend. "I primarily see growth in white-hat SEO. Most are savvy enough to know that they can't afford to be delisted. The industry as a whole is heading toward white-hat SEO." But he also concedes the point that hackers and SEOs "are getting a little more affiliated, and more SEOs are delving into that world."
They've cleaved the other way, crossing into the realm of the illegal to keep the game going. If Google won't let black-hat SEOs build link farms or stuff comments fields with links, then they will exploit legitimate sites and use them as cats' paws in their schemes. Of course, an early target has been .edu domains. "Almost all of the .edu hacks now are for SEO," says RSnake. "Not just a few of the big hacks. I mean almost all of them." Domains with .mil extensions, which also pass "juice" (SEO lingo for tactics that increase Web rankings), are targets now, too.
Primary entries into sites are XSS, SQL injection and FTP vulnerabilities that allow strangers to manipulate the site. Hackers traditionally used those vulnerabilities to insert bots on a site for distributing spam, stealing personal data or some other scam. Now they are being used to stuff links on the page. They hide the links by making them the same color as the background (an old technique for keywords made new) or by simply cloaking them, so that the spiders see them but people do not.
If the site gets good traffic--like Al Gore's ecology blog--those hidden links get good juice. Another scam uses the bots to give redirect commands that send browsers to link farms. Recent headlines illustrate this: "Forth Road Bridge hack redirects to smut bazaar" and "Perl.com sends visitors to porn link farm." Many SEOs said hacking and surreptitious linking are rampant on social networking sites, and blog platforms like WordPress (where Al Gore's blog lived) are under constant attack as hackers look for high-traffic zones to plant their links and their bots.
Another illegal technique a bot might be used for is cookie stuffing. Here's one cookie-stuffing scheme: Around tax time, a hacking SEO uses compromised sites to secretly inject cookies onto the computers of site visitors. On those cookies are referral links to the tax prep websites. If my machine had been stuffed with one of those cookies, the person who put it there would collect a referral fee when I signed up to use one the tax prep sites.
Many experts believe this is only the beginning and that, because there's so much money to be made off the search business model, the techniques will get more sophisticated and far more clever. "From my point of view," says Grossman, "it's just getting started."