CrowdStrike Outage: A ‘Great Company’ That Was ‘Trying To Do The Right Thing’

As one partner tells CRN, the CrowdStrike update that led to a global IT outage was meant to protect against new hacker techniques—and wasn’t merely a new software feature update.

While it may make little difference to the countless people impacted by the massive CrowdStrike-Microsoft outage that began Friday, it wasn’t merely a new software feature update that was responsible for the global disruptions, a solution provider partner told CRN.

“This is being portrayed as a software update,” said Heath Renfrow, co-founder of Fenix24, a CrowdStrike partner based in Chattanooga, Tenn. “It was not.”

Instead, the cybersecurity vendor was ultimately trying to protect customers against increasingly sophisticated hackers when it rolled out the fateful update to Falcon, Renfrow said.

CrowdStrike has disclosed that the update was a sensor configuration update for Falcon aimed to prevent newly observed hacker techniques. The configuration update was designed to halt the use of malicious communications infrastructure, which has been observed to be in use by common command-and-control frameworks as part of cyberattacks, CrowdStrike said.

In other words, the update that sparked an unprecedented Microsoft Windows outage was a response to a potentially dangerous behavioral change by hackers, Renfrow noted. CrowdStrike, which he called a “great company,” was attempting to what was necessary for protecting customers by adjusting to the tactics of threat actors and “how they've been able to bypass the EDR [endpoint detection and response] solutions.”

Attackers are “always evolving,” Renfrow said. “CrowdStrike was trying to do the right thing.”

CrowdStrike’s defective Falcon update led to the “blue screen of death” for Windows systems worldwide on Friday and brought widespread disruptions to air travel, health care, banking and more. Microsoft disclosed Saturday that 8.5 million Windows devices were impacted by CrowdStrike’s update.

The disruptions continued through the weekend and into Monday, notably with Delta canceling hundreds of additional flights scheduled for the day. CRN has reached out to CrowdStrike for comment.

Thousands of flights have been canceled by airlines since the outage began, while some hospitals reported postponing surgeries and some 911 systems were reportedly unavailable on Friday.

“We understand the gravity of this situation and are deeply sorry for the inconvenience and disruption,” CrowdStrike said on its page for the outage incident. “Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”

Helping Customers To Recover

Fenix24 has been among the many CrowdStrike partners that have responded to the outage and sought to help customers recover their Windows devices.

The solution provider has taken a different approach from many others, however, by releasing a free recovery script that aims to help customers get devices restored more easily. The Fenix24 script enables a more-automated approach to recovering Windows and VMware environments, through automatically removing the problematic CrowdStrike files, resetting the blue screens and rebooting the systems, Renfrow said.

The script can ultimately “get things back up and running without having to go machine by machine,” he said.

So far, 200 entities have downloaded the scripts, according to Renfrow.

Fenix24 also directly intervened for one major customer, a large oil refinery, he noted. The team recovered the oil refinery’s virtual machine environments by using a remote management tool to run the Fenix24 automation scripts, Renfrow said.

“They have 100,000 servers in their environment,” he said. “We were able to help them get fully recovered within 24 hours.”

Meanwhile, CrowdStrike has also said it’s working on a “new technique” to expedite recovery even more effectively.

“Together with customers, we tested a new technique to accelerate impacted system remediation,” the company said in a LinkedIn post Sunday. “We’re in the process of operationalizing an opt-in to this technique.”