San Diego Power Outage Shows Importance Of Disaster Recovery

The massive electrical power outage which on Thursday night turned the lights off throughout Southern California, Arizona, and northern Mexico caught solution providers and their customers off-guard, but proper disaster recovery programs appears to have helped many recover quickly.

Even so, the outage once again pushes disaster recovery to the forefront, coming a week after a massive hurricane sent the U.S. Eastern seaboard into turmoil.

The electrical outage was caused by a maintenance worker error in Arizona, but unexpectedly spread to southern California and northern Mexico after safeguards which should have localized the problem failed, resulting in business and school closures and a huge traffic jam in the San Diego, Calif. area.

The severity of the outage also damaged solution providers' data centers and communications capabilities for several hours.

Rich Baldwin, CIO and chief strategy officer at Nth Generation Computing, a San Diego-based solution provider, said his customers had not reported any IT issues by about noon on Friday.

One customer, the University of California at San Diego, actually sent an e-mail to Nth to thank the company for having one of its engineers, Jason Walker, on site to reassure them that their disaster recovery infrastructure was working properly, Baldwin said. Nth had recently implemented the technology for the university, he said.

The power outage was not so kind to Nth's own data center, however.

"We host our web site at a collocation facility, and had no problems with it," he said. "But when the outage occurred, it seemed to spike the power several times, causing our data center's UPS to flip on and off several times. It finally locked into the off position. The batteries were fully charged, and we were running it at only a 50-percent to 60-percent load before the crash."

That UPS failure did cause Nth to suffer minor database corruption, Baldwin said. "But our IT people were on it early Friday morning," he said. "By the time we got into the office, all systems were online, and the databases were up. We didn't miss a beat."

For Bert Shure, account executive in the San Diego office of Denver-based Advanced Systems Group, the main problem was a temporary loss of communications.

Shure said he was not able to get Internet access from his Verizon mobile phone, or Internet or VoIP access from Time Warner Cable. However, he said, his neighbor's AT&T phone service was unaffected.

Shure also said his Google G-mail access disappeared for several hours. "I expected that when the power and Internet came back online, I would get the queued e-mail," he said. "But it took hours to get it."

However, Shure said, customers had not reported any major issues.

The power outage does mean that customers will be getting visits from their local solution providers to discuss beefing up their disaster recovery capabilities.

Baldwin said the outage was a big wakeup call for the importance of disaster recovery, especially after the much more serious problems caused by Hurricane Irene which last week lashed the East Coast, causing sever flooding and wind damage.

"I was talking to people on the East Coast who lost power for a week from Irene," he said. "We had only an 8-hour outage. This shows how dependent we are on power."

The power outage is a chance to talk to customers about upgrading their disaster recovery plans, Baldwin said.

Next: Power Outage A Good Time To Talk Disaster Recovery

"It's the perfect time, while it's fresh in everyone's memory," he said. "Most office buildings don't have power to keep employees working after an outage. They can keep IT systems running. This could get many people looking more towards managed services and cloud computing, especially with so many people now accessing data on iPads, iPhones, and other devices."

Mike Kornblum, president and chief storage architect of MPAK Technology, a San Diego-based solution provider, said, "I am certain that organizations will be looking into UPS systems and generators."

For solution providers, like many others in the San Diego area, the power outage provided a variety of personal inconveniences sprinkled with a rare chance to relax in the late afternoon and early evening.

Kornblum that he has heard of no crazy issues from customers. Instead, it was for him a chance to enjoy a steak and lobster dinner and a round of golf after the power went out at a local casino where his company had installed a video surveillance infrastructure.

Shure said all one can do is make the best of a bad situation. "We used the radio from the survival kits I keep in the trunk of our cars," he said. I had some battery lanterns that were great. Our inside gas stove didn't work, but our outside grill that uses natural gas worked."

Baldwin said his normal 25-minute drive from the office to home took him 2.5 hours, because the traffic signals were without power. Just the one mile from his office to the freeway took 45 minutes, he said.

"But we had candles, batteries in the flashlights, and food in the refrigerator," he said. "When I got home, I saw groups of people sitting outside on their nice chairs with their wine glasses. Nearly all the businesses were out of power, but the liquor store and wine store were open."