VAR Helps eHarmony Find Love With Oracle

Echoing the traditional line used to jilt an ex, eHarmony Vice President of Technology Mark Douglas says the site simply outgrew its old database software. With more than 17 million registered users and new ones arriving at the rate of 10,000 a day, eHarmony needed a data infrastructure with massive scalability. Microsoft's SQL Server couldn't handle the load.

"When I started here just under three years ago, the production database was 150 gigabytes. Now it's 4 terabytes," Douglas said. "It was clear that we were physically outgrowing the database environment. With [SQL Server], if you grow beyond a single server, you have to divide your database or your schema. I said 'let's go with a database that can be clustered.'"

But ripping and replacing the foundational infrastructure of a live, heavily used Web service is no small task. EHarmony, based in Pasadena, Calif., brought in local Oracle consultancy Cloud Creek Systems and spent more than a year building its new data environment and readying for its migration.

Brian Boyd, Cloud Creek's vice president of sales and operations, describes the eHarmony project as one of the most complex his firm has ever tackled. EHarmony has a huge data warehouse, heavy round-the-clock usage, and an exotic technical setup.

An early attempt to deploy eHarmony's production database on Dell servers running Linux ran aground on energy consumption issues and an obscure Linux memory limitation. Instead, eHarmony opted for Windows running on Sun Fire x4600 servers. Getting Oracle to support Windows -- "they support it, they just don't always support it very well," Boyd said -- was one of the project's myriad challenges.

Topping things off was eHarmony's intense expectations. Technical problems that impede the site's operation directly hit eHarmony's bottom line, leaving little room for error. "If someone fubars over there, they get fired," Boyd said.

Cloud Creek and eHarmony prepped rigorously to guard against fubaring the migration. EHarmony heavily scripted the actual migration process and tested it out for 100 days in a row before the cutover date, Dec 5. The company planned a three-hour outage to carry out the switch, a maneuver Douglas likens to "trying to transfer packages between two planes while they're both in flight."

Happily, the transfer went off without a hitch. The biggest hiccup was in communications. Because the site would be down for several hours, everyone at eHarmony knew about the planned migration and awaited word of its outcome. "I sent out a note before we started, but I forgot to send out the second note that it had been successful," Douglas said sheepishly. "Everyone woke up in the morning assuming the world was on fire."

EHarmony's new setup includes Oracle's 10g database with Real Application Clusters (RAC) and Oracle Clusterware. The system easily passed its first major test: Valentine's Day, which traditionally floods eHarmony with users and severely spikes its IT resource demands.

Its new foundation now firmly in place, eHarmony is working with Cloud Creek on database tuning and new IT initiatives like advanced disaster recovery planning. With eHarmony's site clocking as many as 1,200 transactions per second at peak usage, Douglas isn't ruling out the idea of eventually hitting the limits of his new system, but for now, he's content.

"RAC isn't infinitely scalable. You don't want to run a 100-server RAC environment, and we may eventually hit the data-divide issue with Oracle as well. But right now, we're not close to any limits," he said. "If we had not done this, we would have had massive problems."