Disaster Recovery: Bloomsburg VAR Prepared After Helping In Joplin

Lee's Aftermath

Tropical Storm Lee didn't get the national attention that Hurricane Irene did, but don't tell that to the residents of Bloomsburg, Pa., where catastrophic flooding along the Susquehanna River submerged about one-third of the town earlier this month. Bloomsburg is also home to InnoTek Computer Consulting, a solution provider, which faced substantial challenges as InnoTek President Fred Reck and his team helped customers recover.

Luckily, Reck had helped another VAR after the Joplin, Mo., tornado earlier this summer and learned a few lessons about disaster recovery. Here's his story.

Pictured here, is the site of the Bloomsburg Fairgrounds, one of InnoTek's customers, after the flood subsided.

Damaged Drives

George Litterer, an InnoTek technician and service manager, looks to recover data from disk drives damaged in the storm. The VAR was able to get at least some data from about 85 percent of the drives it recovered.

"All our customers were backed up but we got a lot of new customers. We had an assembly line to recover data from flood-damaged laptops," Reck said. "I was shocked at the number of people that came in with equipment dripping in mud and water and with a handful of backup tapes."

Notebook, No More

A damaged notebook sits among a pile of debris after the flood. InnoTek was able to save data from many hard drives, but not the PCs themselves. The scene is similar to what Reck found when he visited Joplin, Mo.-based VAR SNC Squared about two weeks after the tornado demolished SNC Squared's headquarters. Interestingly, Reck and SNC Squared President John Motazedi are friends and had often shared their philosophies on backup and data recovery.

"Bam, then he gets hit. Then this [flood] happened. Before the tornado, we worked on some of our BDR systems together. It really shows the importance of offsite backup," Reck said. "Not just offsite, but offsite to the cloud or geographically-separated backups."

InnoTek's building was not damaged by the flood but the company -- the entire town, in fact -- was without power for several days.

Lessons Learned

Pictured here, Justin Whitehair, a cabling contractor for InnoTek, brings a row boat to a family that could not evacuate the flood. InnoTek also received several flood-damaged servers from the township that had been housed in the blue building.

After witnessing the recovery in Joplin, Reck felt that InnoTek needed to upgrade its own data backup solutions and now keeps data stored in another state, in case of an emergency like the flood.

"It really brought up how important it is not to keep data close to the original server. My data center was OK [after the flood] but we felt that preparation was important,' Reck said. "One of the municipalities came to us with a bag of backup tapes. They took the tapes offsite to the manager's house, but his house flooded."

{C}

The Importance Of Being Backed Up

Reck, with glasses, and Whitehair, make their way through flood-ravaged Bloomsburg.

"The fact that it was a regional and total disaster there [in Joplin] really got us to understand the importance of backup and how my guys need to check it and double check it. I give them hell if they say the backup hasn't worked for a couple days," Reck said. "I tell them they have to understand this is the most important thing we do, backups. It helped us and [Motazedi's] strategy helped us understand that clients' backups are everything. And because of that, some of my customers fared much better than if we hadn't known."

Water Damaged

This blurry image shows a customer's IT equipment as it sat as floodwaters came and then receded at the offices of the Bloomsburg Fair, which took on eight feet of water.

Even though InnoTek's office was not damaged by water, its Kaseya server went down because there was no power, Reck said.

"We learned even more lessons that we need to have all our infrastructure up in the cloud," Reck said. "We were using [Joplin] as an example [of business continuity]. We put what I learned in my newsletter and our Facebook page. In all my sales pitches and backup solutions now, we explain a regional disaster and the importance of backup to cloud."

Cleaning Up

Volunteers work to clean out the offices of the Bloomsburg Fair, an annual event that attracts more than 400,000 visitors and provides an economic lifeblood to the small community of 15,000 people. The flood forced the cancellation of the 157th annual fair this year for the first time.

"Everybody was hoping to get the Fair going. As soon as the waters receded, one of my techs and I tried to get the technology up and running," Reck said. Despite their efforts, the damage to the land and buildings was too expensive to open the fair.

High Water Mark

Someone temporarily posted a new water mark on one of the fair's buildings for the 2011 flood that easily eclipsed the town's previous infamous flood in 1972.

"It's funny, everyone knows Hurricane Agnes [in 1972]. That's a benchmark anywhere. That was last biggest flood," Reck said.

Fiber-Optic Failure

This photo shows some of the fiber-optic infrastructure that supported the Bloomsburg Fair. This section is eight feet off the ground and was submerged under flood water. "We had to replace all that. We'd been working on that all week," Reck said.

InnoTek started in 1999 and has six employees. It offers managed services, systems integration and is starting to get into virtualization and cloud, Reck said.

Sinkhole

The flooding caused many sinkholes around Bloomsburg, including this one near a ticket booth building.

Even though InnoTek's office wasn't damaged, Reck still couldn't get on site because surrounding roads were out. On the third day, Reck found a way in, but the 10-mile drive was extended into a 30-mile drive.

"We were totally isolated from physically accessing the site. It opened my eyes. I learned we need to get ourselves more in the cloud. With something like this, you literally can't get to the site," he said.

More Damage

The fair didn't have full power for nearly two weeks after the flood. Water was also out for a week and the National Guard was on site for a full week, handing out supplies to residents, Reck said.

Closed For Business

The flooding did major damage to the Bloomsburg Fair buildings, including this one, besides destroying IT equipment. InnoTek was able to help get the organization's IT infrastructure back up, which helped employees organize help and communicate, Reck said.

Home Damage

Several houses also fell into sinkholes after the flooding receded. "This one came right off the foundation. All the water just washed the ground away," said Reck.

Prior to the flooding, InnoTek had replicated its servers offsite, but still in the Bloomsburg area. Now Reck said he intends to back up outside the region as another security measure.

"We've got our Exchange servers offsite but all our other servers are virtualized locally. Now we have to find a host we're happy with," he said.

Salvage Mission

Reck and other volunteers pulled all this IT equipment out of the Fair's office.

"We just loaded it onto the back of the truck to transport to our office to try to salvage it," he said. Aside from taking data from the hard drives, not much equipment was able to be saved.

Rescue Center

After the flood, InnoTek opened a technology rescue center where any business that lost its office could use InnoTek computers, Internet and printers. Here, employees from the Bloomsburg Fair look happy to have a place to work.

Battle Plan

Innotek blew up a map of the Bloomsburg Fair's fiber optic locations to help determine the extent of the damage and a plan to replace broken equipment. "This is our war room map. Because everything was so devastated, we've marked it with what's working, what's bad. It's a pretty big campus," Reck said.