CRN Exclusive: Cisco Earns Top Ratings For Response To Spectre, Meltdown Security Crisis

In the days after the Spectre and Meltdown bugs hit the news, Faisal Bhutto, vice president of corporate strategy at Computex, a solution provider in Houston, was struck by the level of confusion in the market.

"The fact is there's no easy fix, and in the first couple of days there was a lot of noise in the market, a lot of confusion in the market," Bhutto said. But as far as Computex's Cisco customers were concerned, the bugs and the strategies for handling them were not a big deal thanks to the networking giant's focused, fast-moving and complete response to the chip vulnerabilities.

"They did a bang-up job, and we were able to get customers aware of what's happening and calm their nerves," Bhutto said.

[Read on: CRN's coverage of the Spectre and Meltdown Response]

In a recent CRN survey of solution providers, Cisco earned top marks among hardware vendors for its response to the Spectre and Meltdown vulnerability discovery; it scored better than bested other industry titans like IBM, HPE, and Dell EMC.

CRN conducted an online poll of 190 members of the CRN Channel Intelligence Council, a panel of solution providers representing the broad channel ecosystem in North America. In the survey, solution providers ranked the vendor responses to the Spectre and Meltdown vulnerability issue on a scale of 1 to 5, with 5 being the top mark, or "excellent."

Cisco scored a mean rating of 3.66 out of five in the survey, which asked solution providers currently involved with the vendor to score vendors' technical, patching, support service and partner communication in the wake of the Spectre and Meltdown bugs. IBM scored 3.58 in the survey, while HPE scored 3.57 and Dell EMC scored 3.57. Lenovo scored 3.55. HP Inc. scored 3.54 and Apple scored 3.47. NetApp scored 3.28.

"The major manufacturers we work with – Cisco, HPE, Dell EMC – they're all world-class organizations, but the engagement we had with [Cisco's] data center team sets them apart," Bhutto said. "When it comes to field engagement, it's not just newsletters and blogs," he said. "That content is there, but at the end of the day, a phone call, a Spark message is more enriching, and that's what stands out."

The Spectre and Meltdown bugs affected CPUs industry-wide, prompting Cisco to issue security advisories covering dozens of products, including UCS servers. Cisco's response, Bhutto said, is helped by the fact that its channel-focused engineers act as specialists rather than generalists.

"We do a lot of UCS business, and you have engineers who are focused on a particular architecture," Bhutto said. "There's a concentrated effort of assigning partner engineers. That's helped, and you could see the results when Spectre and Meltdown happened. For all of our customers who are UCS clients, it was fairly easy."

Close communication and focused engineering expertise have enabled Computex to calm customers and provide clear, comprehensive advice, Bhutto said.

"We held education sessions with each large customer of ours," Bhutto said. "We give them a breakdown, and we assigned three of our experts on data center to put together a very concise summary and present it to customers, and that was a huge help. It's about continual education, making sure customers take care of patches and have a disciplined patching cycle. This is a problem you don't solve overnight, but we had enough data to begin communicating with customers right away."

The relative ease with which Cisco partners and customers have been able to confront Spectre and Meltdown is the result of the vendor's massive efforts behind the scenes. Cisco's Product Security Incident Response Team [PSIRT] has global reach and includes more than 20 incident managers. A minimum of two engineers handles every reported incident. The team supports every single Cisco product.

The team includes analysts whose only job is to look for third-party software vulnerabilities, and that group publishes more than 150 reports every week.

"For Spectre and Meltdown, they affected CPUs for general purpose computing, so UCS devices were affected," said Omar Santos, PSIRT's principal engineer. "We created code even before there was a patch created for those systems. It was a no-brainer that we had to respond and respond very quickly. We worked behind the scenes to make sure whatever microcode we created worked with those systems. There's a lot of interdependencies, a lot of multi-party coordination that we take into consideration."

It's important, Santos said, that whatever tools and education Cisco provides scales easily so customers and partners can quickly and completely consume it. Cisco is also more than willing to get hands-on with partners and customers when a vulnerability surfaces.

"In a lot of cases, we have sent people on site to a partner or to a customer directly to help at least assess the impact," Santos said. "We do have a very strong channel partner community. We try to train the trainer so they are somewhat self-sufficient. We have gone to customers sites and partner sites to help with that. We also try to accelerate and automate a lot of the process. When it comes to large customers, say a service provider that might have 1.5 million devices that might be impacted by one of these things, it's a significant task."

"In some of our strategic partners, we have people who handle that relationship, or a team," Santos said. "We work very closely with resellers. They understand their customers' environments. In a lot of cases, we have experts within Cisco, so if they have to deal with regulations or something like that, they can help. Patch management is always going to be there."

Still, Spectre and Meltdown are a little different in that they involve a complex web of vendors and circumstances. The bugs, Santos said, illuminate one of the industry's key challenges: coordination between technology vendors when incidents occur. Santos has played a key part in developing the OASIS Common Security Advisory Framework, a system that aims to accelerate the assessment of vulnerabilities, to build those assessment capabilities into products, create code to combat those vulnerabilities and hasten disclosure.

"As an industry, we have to evolve a lot better when it comes to multi-vendor coordination," Santos said. "Look at any third-party software vulnerability. It takes effort and time between discovery and when the end user gets a patch. Not just weeks or months, but maybe years. Some may never get a patch. The products that the Ciscos and other vendors have to wait for that patch. Other downstream vendors also have to patch. Weeks become months and years – the way we're sharing information and patching we do need to modernize."

"My team is trying to come up with best practices on how multi-party coordination should take place," Santos said. "We come up with innovative things to do better vulnerability management. We learned about Spectre and Meltdown later in the game, but we were able to alleviate it quickly."