Death Of A Web 2.0 Service

For most of the affected sites, the pain was acute but short-lived. 365 Main had power back on to its customers within 45 minutes, and almost all of its downed customers crawled back online within a few hours. By Wednesday morning, the crash was an unpleasant but dimming memory, a historical footnote serving as yet another reminder that even the "multiple nines" of uptime -- 99.99x to the nth degree -- that hosting providers love to tout leave enough wiggle room for disastrous outages.

But for one affected site, Rojo.com, 365 Main's collapse turned into something more than an annoying but routine service blip. It's become the cataclysmic disaster likely to serve as the final nail in the struggling RSS aggregator's coffin. More than a week after 365 Main's outage, Rojo remains offline, and a sizable group of irate users say they won't be going back to Rojo after its resurrection.

"RIP Rojo," one user summed up on Rojo's status blog, the only functioning part of the Web site.

Rojo's collapse is a cautionary tale for Web 2.0 companies: When your entire service and brand are wrapped up in "always on" availability, downtime has a tipping point, where it transforms from a customer service problem into an unrecoverable death spiral.

"One of the harshest realities of SaaS [software as a service] is that SaaS is actually quite a bit more reliable than traditional software," said Treb Ryan, CEO of OpSource, a Santa Clara, Calif., solution provider that focuses exclusively on SaaS technology and services. "I can almost guarantee you that there are quite a few people out there in the world right now that are not able to get to their Siebel implementations, for some reason or other, because their local networks are having problems. The problem for SaaS providers is that when there's an outage, all of their customers are affected at once. It's disastrous."

Rojo and its parent company, blog technology maker Six Apart, are vague about precisely what technical debacles have kept Rojo offline 10 days after its data center powered back up. A note posted to Rojo's blog on July 27, three days after 35 Main's outage, cited "database corruption" and ongoing efforts to repair damaged data tables.

"This datacenter situation was a perfect storm of failures. We do keep both master and slave database servers to guard against reasonably foreseeable events, but in this case both our master and replicated databases were corrupted," the note said. "Unfortunately, we do not have a usable third backup of the particular table that was affected. This is a situation that 'should' never happen on many levels."

365 Main's crash took down all of Six Apart's Web services, which include blog networks Typepad, Livejournal and Vox, but those sites recovered within the day. Rojo, which Six Apart acquired in Sept. 2006, is the only one to remain offline.

"I think it's going to take the next week before the service is back up," Six Apart spokeswoman Jane Anderson told ChannelWeb, after consulting with the company's engineering team about Rojo's status. No Six Apart executive was available to comment on Rojo's breakdown -- or on how the company will go about rebuilding Rojo users' faith in the service. Anderson emphasized Six Apart's commitment to bringing Rojo back online.

For many users, that's likely to be too little, too late. "I finally have the motivation to make a switch I'd been planning anyway," said Stephanie Johnson, who began using Rojo last year. Periodic short outages had her reconsidering her Rojo use, and she began experimenting with rival aggregator Google Reader. This week's prolonged Rojo crash has her ready to migrate for good.

"Service outages of any flavor are inevitable. Service outages of this duration deserve thorough explanations. Rojo has failed to communicate with users," Johnson said. "When they are back, I'll be logging in just long enough to get an OPML file of my subscriptions so I can take it to a different aggregator."

NEXT: Lack of communication makes matters worse

Frustration with Rojo's sparse communication about the crash and its revival plans are a common theme in user complaints. Aside from two terse blog posts, one last Friday and one this morning, Rojo's administrators haven't offered any comment on its travails and haven't given users a timeline for when it expects to be back online. But they have updated Rojo's blog with chatty "week in review" posts rounding up commentary from around the blogosphere -- commentary touting the very posts that Rojo users are missing while they're cut off from their feeds.

"The almost complete lack of communication (combined with the fact that they updated their blog with a marketing post while the product is still down!!) is amazing," user Noah Tutak said via e-mail. "If another product provided a way for me to easily browse through a large number of feeds through some sort of social recommendation system I would jump in a minute."

Rojo.com's clean visual interface and community-oriented blog post ranking features are two of its key differentiation points in a crowded field. Launched in mid-2005, Rojo initially drew strongly positive reviews and word-of-mouth endorsements. Business Week chose it that year for its "Best of the Web" roundup, and the Wall Street Journal singled Rojo out as unique among RSS readers in the depth of its personalization features.

But Rojo's technology aged, without much visible development after the Six Apart acquisition, and its rivals gained ground. Google Reader, in particular, launched in late 2005 and has steadily improved, gradually siphoning users away from other RSS feed aggregators.

GorillaSushi.com blogger Jason Phillips had been a Rojo evangelist since adopting it last year, but now, he too is ready to migrate to Google.

"Every time I would write about RSS feeds or blog-reading tips I would always recommend Rojo to my readers. The typical response was 'you should try Google's Reader,' and that's exactly what I've now done," Phillips said. "I won't be going back to Rojo. I wanted to switch after only a day or two [after the blackout] but I had not made a backup of my OPML files. I thinking backing up OPML files is the kind of thing that most people only consider once it's too late."

Disaster recovery planning is also a precaution too many companies consider only after the fact, according to Richard Heitmann, vice president of product management for EVault, a Seagate Technology subsidiary that specializes in data backup technology and services.

Disaster recovery, or "business continuity," as vendors in the space like to call it, is expensive. Backups of any sort, much less multi-site backups with geographical redundancy, simply don't come cheap. But as vendors like EVault point out, recovering from a catastrophe can be even more painful than budgeting for precautionary measures. Crashes like 365 Main's widely publicized outage tend to spur companies into moving one step further along on the disaster-planning road map, Heitmann said.

"When disasters happen, some companies realize that they need to get out there and put that disaster recovery plan in place. Others are reminded that a disaster recovery plan is only good if it works. They're motivated to actually go out and test their plans," Heitmann said. "What we're finding is that with the recent disasters and system outages, we see a much greater awareness. People shift along the adoption curve."

Some also learn from a mistake that history shows being repeated time and again: keeping users in the dark. Numerous Rojo users expressed almost as much irritation with the company's continuing lack of communication as with the outage itself.

The SaaS industry's most notable poster child, Salesforce.com, learned the hard way about the importance of a steady flow of real-time information. In late 2005, it suffered a series of outages that dinged its reputation as a reliable provider of "on-demand" service. Customers complained loudly about both the downtime and Salesforce.com's silence. Salesforce.com was forced to post a "mea culpa" message apologizing for the crashes and for its spotty communication, and within weeks it launched a portal -- http://trust.salesforce.com/ -- to track its availability. If it suffers future outages, it now has infrastructure in place to keep customers up to date on its status and repair efforts.

Creating that communication infrastructure is a key part of SaaS disaster planning, OpSource's Ryan said.

"It's one of the biggest mistakes I think people make, not communicating," he said. "Customers don't want you to hide when you're out."

For Rojo, such words of wisdom likely come too late. With its downtime heading into the second week, users are giving up: "For all I know they've skipped town," Johnson said. GorillaSushi's Phillips has seen his blog traffic numbers take a hit since Rojo's crash, since he's no longer following and interacting with many of the other blogs he uses Rojo to track.

"My Rojo list of feeds represents a year's worth of exploration and discovery to me," Phillips said.

Now, it represents only a list of feeds he plans to grab as soon as he's able to export to a different aggregator.