Spectre, Meltdown Update: VMware Retracts Faulty Intel Firmware Patches For Chip Vulnerabilities

A bug in Intel's firmware update for Meltdown and Spectre that manifests itself when running virtual machines has prompted VMware to rollback a recently issued security patching recommendation. This adds yet another wrinkle of complexity for solution providers looking to protect their customers from the two side-channel vulnerabilities.

Customers alerted Intel that upgraded systems powered by some older Intel Haswell and Broadwell processors were experiencing unexpected reboots when running the Intel firmware upgrade.

VMware had recently pushed out the Intel microcode upgrades for the physical hardware, as a courtesy to its customers, alongside its own patches for ESX hypervisors—a mitigation framework the virtualization vendor recommended in its latest security advisory on the side-channel vulnerabilities.

[Related: Red Hat Researchers: Spectre Chip Vulnerability Likely Worse For VMs Than Containers]

On Monday, once Intel alerted VMware of the problem discovered in the field, VMware advised partners to no longer install those patches in tandem. VMware instead referred them to an earlier security advisory document that referenced patch releases not including the code from Intel.

For customers that already implemented both sets of patches, VMware shared a config line shutting off Intel's speculative execution control mechanism which implements the functionality at the heart of the problem.

"The issue can occur when the speculative execution control is actually used within a virtual machine by a patched OS," said the knowledge base article from VMware alerting users of the new problem.

VMware told customers the Intel upgrade shouldn't be installed at all for vSphere environments until the chipmaker fixes the problem.

"At this point, it has been recommended that VMware remove exposure of the speculative-execution mechanism to virtual machines on ESXi hosts using the affected Intel processors until Intel provides new microcode at a later date," the statement said.

VMware has released a single hypervisor patch that addresses variant 2 of Spectre—an attack mechanism called branch target injection. The other Spectre variant and the Meltdown vulnerability unique to Intel chips don't pose risks through the hypervisor. VMware intended the microcode patch to allow guest operating systems running in virtual machines to also mitigate against the Spectre variant.

The reboot issue stemming from running Intel's microcode went unnoticed when those updates were developed and tested in the lab. But once released, "sightings"—an Intel term for anomalous chip behavior observed in the field—triggered closer examination, leading to the rollback for systems with affected processors.

"VMware is working closely with Intel and the industry to come to a quick resolution of this Intel microcode issue and provide an update to our customers as soon as possible," the VMware article said.

The latest bug illustrates the complex challenges faced by solution providers trying to protect their customers from the chip vulnerabilities that are causing turmoil across the industry. It suggests headaches associated with Meltdown and Spectre are unlikely to be resolved anytime soon.

Jason Malacko, the manager for security practice relationships and offerings at Logicalis, a global VMware partner, has been experimenting with patch combinations in a lab environment as his firm prepares to deploy those mitigations into production vSphere environments, both in its own data centers and those of customers.

"As an MSP, it's frustrating," he said of the complex issues around security and performance caused by the chip vulnerabilities and their fixes. "But this is an industry-wide issue. It's a microcode issue, hypervisor issue, on top of that it’s a guest [OS] issue. You have to look at this from so many aspects."

Large solution providers like Logicalis won't throw patches into production environments, even when pushed out by prestigious software vendors before they put them through extensive testing. Customer workloads are often highly customized, and partners cannot assume patches will work as the vendors' intend, Malacko told CRN.

While he hasn't seen the latest Intel microcode bug manifest itself in the lab setting, the document from VMware provides a useful warning as Logicalis explores how the tapestry of patches will fare in real-world environments, he said.

Meltdown and Spectre represent "one of the most complex scenarios" as far as updating systems, he said. But all the anxiety around those potential attacks shouldn't distract attention from the 14,000 other documented vulnerabilities.

As to VMware's release and then retraction of the Intel patches, that kind of back-and-forth isn't uncommon in scenarios like the one the industry is currently facing.

"There are all kinds of vulnerabilities, and it takes time to work through them, and the first fix isn't always the final fix," he said. That's all part of the job of being a solution provider, he said.