Storm Grinds IT Operations For Bay Area Businesses To A Halt

A massive storm Thursday brought many Bay Area businesses in Northern California to a standstill due to insufficient disaster recovery protection.

Roughly 70 percent of Berkeley-based Endsight's customers were forced to shut down when the lights went out because they had failed to purchase a hot site, which would have allowed their networks to continue running on a private cloud with virtually no interruption.

"Sometimes people have to get stung before they're willing to spend the money," said Endsight CEO Mike Chaput.

[Related: Bay Area Earthquake Creates Ripple Of Outages, Downtime]

The biggest storm to hit Northern California in years dumped anywhere from 2.5 inches to 6 inches of rain on the region and left more than 250,000 customers in the dark. The floodwaters forced many roads to close and more than 200 San Francisco International Airport flights to be cancelled.

Roughly 75 percent of Chaput's customers take advantage of the cloud, but most only use it for data backup, which only would come in handy if information is lost. Generators also are rare among Endsight's customer because of the emissions it would cause in most office buildings.

"These natural disasters are a great opportunity for IT services firms," he said. "They prove the impact of a proactive strategy."

The storm, however, is more likely to result in brief power outages rather than any real damage for customers of Clare Computer Solutions of Sam Ramon, Calif., said Vice President of Marketing Bruce Campbell.

"Most of our install base has been in pretty good shape," Campbell said.

Campbell, who said he sees customers that keep servers on the floor, urges them to elevate IT infrastructures to avoid any damage from surface runoff or pipes that burst.

Chaput said things went off without a hitch for the 30 percent of their clients with a full cloud package, which allows them to operate all systems off of Endsight's private cloud. Customers of Clare Computer Solutions keep most of the assets they need to complete day-to-day work in the cloud, Campbell said.

"Doing this is going to save money for everybody in the long run," Campbell said.

Clare also sets up its customers so that lighter equipment will rely on battery power in the event of a power outage while heavier equipment will transition to an uninterruptable power supply, which can keep computers running.

Endsight's offices were without power from about 2 a.m. to 10 a.m. Thursday, but the solution provider was unaffected because all the infrastructure needed to run its business is kept on a private cloud.

"We drink our own Kool-Aid, so to speak," Chaput said.

Endsight keeps its systems and servers in flood-protected data centers with generators and around-the-clock security, he said.

Customers who lost power also require help when power is restored, Chaput said. The UPS automatically shuts the servers down once there's no electricity without any type of power surge.

But when the lights come back on, Endsight's team helps customers manually reboot the servers. Chaput said they can almost always send remote "power on" instructions without resorting to a site visit.

Although flooding wasn't a huge issue for Endsight's customers as of Thursday, Chaput said he was aware that there was more rain in the forecast.

"I'm not declaring a victory yet," he said.

PUBLISHED DEC. 12, 2014