‘Quick Execution’ By MSPs Made 'All The Difference’ For Some In The CrowdStrike Global Outage

'While some of our peers were down for days, iCorps had us back on track within hours,' says a manufacturing company CEO in an email describing the scenario. 'Their proactive approach and quick execution made all the difference.’

When the global CrowdStrike outage hit on July 19, MSP iCorps Technology Inc.- which monitors systems for its clients 24/7, 365 days a year, alerted customers before they even knew systems were down and had a workaround in place by 6 a.m. EST.

The “all hands on deck” response from Woburn, Mass.-headquartered iCorps, No. 486 on the CRN SP500, got about 2,500 systems quickly back up and running.

The fast-moving response earned high praise from a manufacturing company that was able to get its production line with 500 employees working with just two-and-a-half hours of lost time.

“iCorps stepped in swiftly, notifying us by 6:30 AM with a workable solution,” said a manufacturing company CEO in an email describing the scenario. “Thanks to their rapid response and expertise, we were back to normal operations by 10:00 AM. While some of our peers were down for days, iCorps had us back on track within hours. Their proactive approach and quick execution made all the difference."

iCorps Technology President and CEO Michael Hadley (pictured above) said the CrowdStrike outage has helped to shine the spotlight on the all-important role of MSPs as trusted advisors to prevent and then remediate outages or security issues in an age when IT is the engine that makes businesses of all sizes run.

In fact, Hadley said, the outage points to the tight “partnership” between MSPs and their customers. “It’s a partnership – not a vendor-client relationship,” he said. “We are always looking out for our customer’s best interest. Their success is our success.”

iCorps is one of many MSPs at the Xchange solution provider conference in San Antonio this week that pointed to the CrowdStrike outage as a pivotal moment that will have far-reaching implications from just how much access vendors like CrowdStrike have to the Microsoft kernel to patch management processes to new remediation plans for MSPs on how to respond to such an outage. They said the outage is acting as a door opener for different discussions on IT risk and remediation.

CrowdStrike, for its part, has said it will provide “additional testing” prior to production employment for the type of update that led to a “blue screen of death” for 8.5 million Windows devices.

CrowdStrike’s probe into the cause of the outage also found that such updates “should be deployed in a staged rollout.” As a result, the vendor said that its content configuration system “has been updated with additional deployment layers and acceptance” checks.

“This incident shows that if we are not helping customers as their first line of defense they are vulnerable to a number of things even accidents,” said Don Monistere, president and CEO of General Informatics, a Baton Rouge, La.-based solution provider that sprung into action with boots on the ground to help bring 142 systems back online for municipalities hit by the outage.

“We had police officers with systems that were not working in their cars,” said General Informatics President and CEO Don Monistere. “It was an eye-opener!”

The outage even has some customers unfortunately asking whether they need certain security functionality, said Monistere. “That has actually come up,” he said.

LAN Infotech, a Fort Lauderdale, Fla MSP, helped a hotel chain get back up and running by sharing the remediation solution to with customers, said LAN Infotech President and CEO Michael Goldstein.

The outage points to just how vulnerable MSPs and customers are with regard to vendor updates in a market where there is a constant barrage of security threats, said Goldstein. “It’s kind of scary,” he said. “This is the first time we have seen a meltdown pretty much across the world for a big vendor. I can’t remember something like this happening.”

Goldstein said U.S. customers were fortunate that the outage started across the globe and then made its way to the United States. “By the time we got up, CrowdStrike was already working on it,” he said. “If we were first responders it would have been a different story.”

Goldstein expects vendors to do more QA (Quality Assurance) testing on updates. Furthermore, he expects more customers to look at VDI solutions that helped bring systems back online quickly like Azure Virtual Desktop. “If you were on a virtual environment you could have reverted back very quickly,” he said. “This incident is leading to different discussions with customers.”

Many customers are realizing that with employees continuing to work from home it makes sense to protect their IT environments with Azure Virtual Desktop. “Customers don’t want to rely on home IT equipment so we are seeing a lot of people move to a virtual desktop,” he said. “The CrowdStrike outage has brought this discussion of virtual desktop further up on the priority list.”

John Krikke, a partner and vice president for Onward Computer Systems, a Burlington, Ontario-based MSP, also expects the outage to cause more customers to look at Azure Virtual Desktop.

“The CrowdStrike outage is a reason to have the conversation,” he said. “It’s a door opener that is changing the conversation about what we are managing.”

Krikke said he was fortunate that he only had one system to remediate out of 700 endpoints. “We had to manually remediate, go on site and clean it up,” he said. “Thank goodness it was one!”

Krikke said he sees the outage as a call to action for MSPs to put together response plans in the case of another outage sparked by a vendor automatic patch update.

“As MSPs we need to look at how can we reasonably respond, what can we do, we obviously can’t have standby equipment for every machine,” he said. “It is all about how do identify, triage and get critical systems up and running.”

In comments posted on August 6 on CrowdStrike’s Remediation and Guidance Hub page, CrowdStrike CEO George Kurtz thanked partners and customers who “mobilized immediately to restore systems” after the outage. “We could not have accomplished so much, so quickly, without your collaboration,” Kurtz said.

LAN Infotech’s Goldstein, for his part, said ultimately there are many lessons to be learned that need to be taken forward. “From disasters like this we have to learn something and become smarter,” he said.