‘Nightmare Scenario:’ Linux Supply Chain Hack Was An Inside Job

The industry appears to have been spared from potentially massive impacts of the open-source attack, after the insertion of a backdoor by a project maintainer was quickly caught by a Microsoft engineer.

In a software supply chain hack described as a “nightmare scenario” by multiple experts, a contributor to an open-source project used by most Linux distributions was responsible for the breach that was nearly disastrous for the IT industry and customers.

Instead, thanks to the efforts of a Microsoft engineer, the backdoor inserted by the insider was discovered Friday before the compromised software could be distributed broadly.

“This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario,” wrote Filippo Valsorda, a cryptography engineer and professional maintainer of open-source projects, in a post.

Red Hat and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) warned Friday that the two latest versions of XZ Utils, a widely used set of data compression tools and libraries in Linux distributions, were found to have been compromised.

On Saturday, the original maintainer of the XZ Utils project, Lasse Collins, disclosed that another maintainer of the project—known by the name “Jia Tan”—had inserted the malicious code. The backdoored software was “created and signed by Jia Tan,” Collins wrote.

Jia Tan, whose handle was JiaT75, had begun contributing to XZ Utils at least two years ago, according to Sam James, a contributor to Linux projects, in a GitHub post. Jia Tan “gained commit access, and then release manager rights, about 1.5 years ago,” James wrote.

A Microsoft engineer, Andres Freund, said in a post Friday that he discovered the vulnerability after noticing “odd” behavior in installations of Debian, a popular Linux distribution—including that logins were taking longer and using more CPU than usual.

Security researchers have credited Freund with going the extra mile to hunt down the issue, ultimately revealing the backdoor in the software.

Freund made the critical discovery through “an incredible attention to detail,” which “led to him finding one of the most public displays of a motivated attacker to date,” wrote Michael Skelton, vice president of security operations and hacker success at Bugcrowd, in a post.

Red Hat confirmed Friday that the affected XZ Utils software had not been widely utilized in Linux distributions. The exceptions were the Linux distributions that typically bring in new packages as soon as they are available, such as Fedora Rawhide and Debian unstable, which were impacted by the hack.

The implanted code is found in versions 5.6.0 and 5.6.1 of the XZ Utils libraries, according to IBM-owned Red Hat. The libraries “contain malicious code that appears to be intended to allow unauthorized access,” Red Hat said in its advisory Friday.

In a comment provided to CRN, Red Hat’s Vincent Danen indicated that the worst potential outcomes from the compromise would seem to have been averted.

“Red Hat, along with CISA and other Linux distributions, were able to identify, assess and help remediate this potential threat before it posed a significant risk to the broader Linux community,” said Danen, vice president of Red Hat product security.

Ultimately, “the malicious code found in the latest versions of the xz libraries show just how critical it is to have a vigilant and veteran Linux security team monitoring software supply chain channels,” he said.

Supply chain compromises rank high on the list of most-feared cyberattacks. Past incidents have included some of the most widely felt cyberattacks to date, including the SolarWinds supply chain attack of 2020 and Kaseya VSA attack of 2021. More recently, communications software maker 3CX suffered a supply chain compromise in March 2023.

The attack against SolarWinds, in particular, was a “big wakeup call for those types of attacks,” said Hooman Mohajeri, vice president of security services at BlueAlly, a Cary, N.C.-based MSSP.

Software supply chain attacks are especially “harder to uncover” because so many applications today leverage third-party libraries, Mohajeri said.

“You have third-party libraries, and sometimes they’re using other third-party libraries,” he said. “It’s one of the challenges we see in software supply chain issues.”

For the Linux software supply chain hack, the incident represents exactly the sort of risk that experts have been warning about, said Ian Coldwater, Kubernetes Security co-chair, in a post on X.

“This upstream supply chain security attack is the kind of nightmare scenario that has gotten people describing it called hysterical for years,” Coldwater wrote.

Red Hat said in its advisory Friday that users of the Fedora Rawhide distribution of Linux should “immediately stop” all usage. “Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed,” the company said.

“Under the right circumstances this interference could potentially enable a malicious actor to break [Secure Shell Daemon] authentication and gain unauthorized access to the entire system remotely,” Red Hat wrote.

Red Hat Enterprise Linux is not affected, the company said.

In its advisory Friday, CISA said it “recommends developers and users to downgrade XZ Utils to an uncompromised version—such as XZ Utils 5.4.6 Stable—hunt for any malicious activity and report any positive findings to CISA.”

The vulnerability created through the supply chain hack is being tracked at CVE-2024-3094.