Cutbacks Could Be Causing IT Outages

When eBay Inc.’s PayPal unit suffered a worldwide outage early this month, Sailrite Enterprises Inc., a sailing supply company based in Churubusco, Ind., lost its critical customer payment services for six hours.

The next day, Aug. 4, PayPal’s services failed Sailrite again — this time for about an hour, according to Matt Grant, a vice president at Sailrite. Grant was not amused. He posted a blunt message on PayPal’s blog site: “This is not acceptable.”

In an e-mail, San Jose-based PayPal blamed the outage on a problem with a “back-end router” that was complicated by a failure in the company’s redundancy measures.

The PayPal electronic payment system is one of many Internet-based services that have been hit with outages. And based on news reports, the number of such incidents appears to have been increasing in recent months, analysts said. They cited shutdowns of the Google Apps software hosted by Google Inc., outages at data centers run by Rackspace Hosting Inc. and a distributed denial-of-service attack on Twitter.

Observers pointed to several possible reasons for the apparent uptick in online outages, including IT budget and personnel cutbacks, increasing corporate dependence on hosted applications — and bad luck.

Kurt Roemer, chief security strategist at Citrix Systems Inc. in Fort Lauderdale, Fla., said he wonders whether a two-hour shutdown of Cisco Systems Inc.’s Web site this month “would [have] happened a few years ago… when they had multiple people checking every single change.” Cisco blamed the outage on human error.

IT staff cuts spurred by the economy are likely to continue throughout the remainder of the year. According to a survey of 300 IT center managers last year by the Association for Computer Operations Management, half of all data centers were planning to cut 2009 budgets by an average of 15%. Respondents at 14% of those companies said the cuts would include layoffs of IT staffers.

Ken Brill, executive director of Uptime Institute Inc., a data center engineering and consulting firm, said such budget and personnel cutbacks can prove disastrous to IT. “We’re not doing the maintenance we should be doing, and when you don’t do maintenance, you increase the probability of catastrophic failure,” he said.

Brill added that energy-efficiency efforts may be prompting data centers to cut back on redundant equipment and run their systems harder, exposing equipment flaws.

Leslie Daigle, chief technology officer of the Internet Society standards group, said that the online service business will probably survive despite periodic outages. “The Internet is in a state of constant evolution, and that really does provide its overall resilience,” she said.

And some IT managers say that recent outages aren’t changing their minds about using hosted services.

For example, Ted Maulucci, CIO at real estate developer Tridel Corp. in Toronto, defended the durability of hosted data center systems, despite the fact that his provider had suffered an outage. “Five years ago, it was not uncommon to experience the odd interruption, whereas today it has been pretty rock-solid, other than the major failure that happened,” he said.

Neal Puff, CIO for Yuma County, Ariz., agreed, even though the county moved its ERP system from a cloud-based provider back in-house last year. Puff said he still believes that “a well-run hosted solution with reliable connections will perform well and be as reliable as any well-run in-house system.”

This version of this story originally appeared in Computerworld ‘s print edition. It’s based on an article that originally appeared on Computerworld.com.