CrowdStrike Chief Security Officer: ‘To Say We’re Devastated Is A Huge Understatement’

‘We let down the very people we committed to protect,’ wrote Shawn Henry, longtime CSO at CrowdStrike, in a LinkedIn post.

The massive global IT outage caused by a faulty CrowdStrike update has been a “gut punch” for the company, though it “pales in comparison” to what customers and partners have been going through, CrowdStrike Chief Security Officer Shawn Henry wrote in a LinkedIn post.

“On Friday we failed you, and for that I'm deeply sorry,” Henry wrote.

CrowdStrike’s defective Falcon update led to the “blue screen of death” for Windows systems worldwide on Friday and brought widespread disruptions to air travel, health care, banking and more. Microsoft disclosed Saturday that 8.5 million Windows devices were impacted by CrowdStrike’s update.

The disruptions continued through the weekend and into Monday, notably with Delta canceling hundreds of additional flights scheduled for the day.

‘Most Challenging 48 Hours’

In the LinkedIn post, Henry said that CrowdStrike’s mission has always been “‘to protect good people from bad things,’ and we've been very successful for more than a decade.”

“On Friday, though, we failed. The past two days have been the most challenging 48 hours for me over 12+ years,” wrote Henry, who was formerly executive assistant director of the FBI before joining CrowdStrike as CSO in 2012.

“The confidence we built in drips over the years was lost in buckets within hours, and it was a gut punch,” he wrote. “But this pales in comparison to the pain we've caused our customers and our partners.”

Ultimately, “we let down the very people we committed to protect, and to say we’re devastated is a huge understatement,” Henry wrote.

Experts have called it the largest IT outage of all time. CRN has reached out to CrowdStrike for comment.

In his post, Henry noted that CrowdStrike has had thousands of team members working around the clock since the outage began, which he expects “will continue for the immediate future.”

“I know I speak for the women and men of CrowdStrike when I say thank you to every customer and partner who has also been working around the clock. You are the real heroes in all of this,” Henry wrote. “We are committed to re-earning your trust by delivering the protection you need to disrupt the adversaries targeting you. Despite this setback, the mission endures.”

Expediting Recovery

CrowdStrike and Microsoft have sought to help expedite the recovery process for the millions of affected Windows devices.

Over the weekend, CrowdStrike posted a “Remediation and Guidance Hub” that aims to assist with recovering from the outage, including through providing technical details and guidance on key areas of focus for IT administrators.

CrowdStrike has also said it’s working on a “new technique” to expedite recovery even more effectively.

“Together with customers, we tested a new technique to accelerate impacted system remediation,” the company said in a LinkedIn post Sunday. “We’re in the process of operationalizing an opt-in to this technique.”

Microsoft, meanwhile, released a free tool Sunday to help clients recover from the outage, enabling admins to more quickly recover Windows devices using a more automated approach.

In an update late Friday evening, CrowdStrike identified a “logic error” as the culprit in the Microsoft outage. The programming error was triggered by a sensor configuration update to Falcon.

For a still-unknown reason, “this configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems,” the company said.