FireEye, NSS Labs Continue To Trade Barbs Over Testing Report Credibility

Security vendor FireEye and independent testing firm NSS Labs continue to trade blows over the legitimacy of a report that aimed to gauge the performance, security efficacy and cost of ownership of so-called breach detection vendors. Meanwhile, solution providers and systems integrators tell CRN their larger clients review third-party testing but place more weight behind an evaluation of the networking boxes in the environment they are trying to protect.

FireEye, which scored "below average" in the NSS Labs comparative group product test for breach detection systems, is calling the testing severely flawed. In an interview with CRN, Manish Gupta, FireEye's senior vice president of products, said NSS Labs ignored the stated purpose of the company's advanced threat appliance, which is to detect custom malware used in targeted attacks.

NSS Labs was testing with known malware samples, Gupta said. The samples had been known to Virus Total, a Google-owned service that tests whether malicious software can be detected by antivirus scanners. All of the malware used in the testing had been known for months or years, Gupta said.

[Related: Palo Alto Networks, FireEye Criticize NSS Labs; Testing Firm Defends Itself ]

"We are a vendor that specializes in advanced attack detection, not in detecting known, stale samples," Gupta said. "We ran their malware samples in our lab and detected every single one of them."

A valid test would have used a zero-day exploit to evaluate the detection capabilities of the appliances or, at a minimum, the testing could have been done in a live, customer environment, Gupta said. FireEye executives believe that data from a test conducted last year on its appliance was used for the latest report against firms that were tested more recently.

"They took the same malware sample set and used it again this year," Gupta said. "In the world of security and especially the world we specialize in, 62 percent of attacks we never see again; that is how targeted some attacks are."

NSS Labs issued the breach detection systems test April 2. It evaluated security appliances from AhnLab, Fidelis, FireEye, Fortinet, Sourcefire (Cisco) and Trend Micro to measure their security effectiveness, relative performance and total cost of ownership. NSS Labs CEO Vikram Phatak told CRN that the firm aims to provide businesses evaluating appliances with a relative value of security investment options.

In a response to FireEye's criticism, NSS Labs said it does its testing in a live environment. "All exploits that are run and all malware that is dropped are live on the Internet at the time of the test," the company stated in a blog post addressing FireEye's points. NSS tests found FireEye detected 94.5 percent of attacks, NSS Labs said. The appliance didn't have 64-bit support, causing it to perform more poorly than other vendor appliances tested by the firm, according to the blog post.

Systems integrators deploying FireEye appliances say potential clients nearly always see the benefits when it is demoed in their environment. FireEye is a unique technology that is designed to be efficient at detecting sophisticated attacks, said Peter Humphries, a principal at Burlington, Ontario-based networking and security services provider SecureSense, a FireEye partner. Humphries said the appliance is meant to be used with a holistic security strategy. Businesses must combine the advanced threat detection with prevention technologies and ensure security policy and process best practices are in use. Nearly 90 percent of SecureSense clients put the FireEye appliance inline, he said, indicating that the appliance can block attacks from entering the network.

"We always position it as a layered approach," Humphries said. "We have successfully deployed FireEye in many organizations in which we baked off with companies tested by NSS Labs, and in many cases we are winning with FireEye."

Fortinet, which received high marks in the test along with Trend Micro, Fidelis and Sourcefire (Cisco), has licensed the test results for use in a variety of marketing campaigns and customer literature. Fortinet introduced a new sandboxing appliance last year to detect advanced malware and has been quietly bringing it to market.

Justin Kallhoff, CEO of Lincoln, Neb.-based Infogressive, a Fortinet partner, said marketing isn't as valuable as having a genuine conversation with a potential client about the issues it is truly trying to address. Test results are difficult to apply because every business has a different risk tolerance and security program maturity level, he said. Strategies need to be based on an organization's overall security posture.

"There's no silver bullet. For us it is more about focusing on the reality of the situation," Kallhoff said. "Every business is going to have infected endpoints so the idea is to lower the window of finding them and in turn lower the impact when it happens."

How an organization configures and manages the new line of appliances is at the core of how effective they can be, said SecureSense's Humphries. For example, a FireEye appliance alerted on malware used in the massive Target breach, weeks before any credit card data was ever stolen. Target failed to take action and experts say it's indicative of a breakdown in its incident response processes.

Third-party tests may provide important data on detection rates, but security experts advise clients to take a thorough approach. Large enterprises and businesses in heavily regulated industries are adopting FireEye and technologies designed to disrupt advanced attacks, said Jon Oltsik, senior principal analyst at Enterprise Strategy Group. Oltsik said most of the companies already have mature security programs, but incidents will continue to happen because no security program is perfect.

"When these technologies see malware that is the start of your work, that's not the end of your work," Oltsik said. "If something is generating an alert, you don't know how many systems are impacted, what has changed on those systems or how long an attacker has been there. Incident response is all part of that."

PUBLISHED APRIL 9, 2014