Let's Can Spam: Round II

The Test Center has been immersed in spam for the past few weeks as we work our way through all the products that are participating in the Let's Can Spam roundup. Our inaugural summary of results looked at the MX Logic Email Defense Service MX Ultimate Access, Sendio ICE Box eMail Integrity Services Appliance, and Sophos ES1000. This week, we present the results of two more appliances: Symantec Mail Security 8300 Series and eSoft Threatwall 250. (Click back next week for more.)

As in the initial set of comparisons, the products were evaluated from three viewpoints: the solution provider, the system administrator at the customer site, and the end-user. Over a two week-period, all inbound traffic was passed through the test product before reaching the mail server. Each system was given time to "learn" the mail before assessing its accuracy. The mail server, a Linux machine running exim, is a production server handling approximately 14000 messages each day. The series kickoff has more details on the test methodology.

For the most part, every single product reviewed so far blocked at least 90 percent of spam. Reviewers focused on how the products vary in handling the remaining 10 percent -- the ones that are borderline spam, mailing lists and application-generated alerts, and image/attachment spam.

Symantec Mail Security 8300 Series " SMS 8360

Symantec sent the SMS 8360, a 1U appliance that also takes a multi-layer approach to anti-spam. Symantec Security Response and the Email Security Group update the appliance which downloads the latest definitions every five to ten minutes. The SMS 8360 filters on reputation, URL, language, and content. Heuristic and signature-based scanning is also performed, as well as looking at whitelists and blacklists. Like MX Logic, Symantec claims to trap PDF spam.

Deployment was very similar to the Sophos appliance, although it wasn't as quick. The interface was very straightforward, but the system offered more choices that the other appliances didn't and more detailed options for what to do with filtered messages. In fact, Symantec offered the most comprehensive solution at this point, with email firewall, anti-spam, anti-virus, IM security, content control, and a way to define email policies.

Under testing, the SMS 8360 took longer to approach accuracy rates similar to the Sophos ES1000. The ES1000 correctly identified some messages as spam that SMS 8360 did not. The two domains that were spam-free under the Sophos ES1000 averaged two spam messages in the inbox every three days under the SMS 8360. The other domains reported about ten to twelve spam messages appearing in the inbox, and about ten to fifteen messages in quarantine, with two or three valid messages. The rates of false negatives and positives dwindled as the appliance learned, and the spam that slipped through was never unmanageable. The SMS 8360 suffered in comparison with the Sophos ES1000 mainly because it took longer to learn.

Messages that may or may not be spam are stored online in user-specific quarantines, where users can periodically check for false positives.

The SMS8360 approaches management and reporting a little differently, putting in management options as links on the left, and reporting options on the top. The dashboard shows graphs with breakdowns, identifying the types of threats the messages had. It is easy to tell how many were spam, viruses, compliance violations, invalid recipients, and poor reputation. The SMS also tracks multi-threat messages and single-threat messages. On an average day towards the end of testing, the SMS identified 12,000 "threats" of which only 16 had multiple threats. Detailed information is available under the Virus, Spam, and Compliance tabs. The dashboard also provided information on found viruses, any compliance violations, and a "threat level" as defined by Symantec for that day.

The SMS 8360 is designed for large enterprises, and it shows, with two Xeon 5130 Dual-Core 2.0 Ghertz processors, 4 Gbytes of memory, two 146 Gbyte serial-attach SCSI disk drives in RAID 1, dual embedded Gigabit Ethernet, a DVD drive, and dual power supplies.

The Symantec Partner Program features four membership levels: registered, silver, gold, and platinum. Partners qualify for a level based on previous and current contributions and investments. Partners have access to pre-release and beta programs, opportunity registration program, post-sales support, internal use licenses, dedicated account representatives, and sales and marketing tools.

Partners can expect margins as well as other revenue opportunities such as highly competitive rebate programs, a cash rebate with the deal registration program, and rebates through the Aspire Rebate Program. Sales training is available on the Web using PartnerNet.

eSoft ThreatWall 250 with the Email ThreatPak

eSoft sent the ThreatWall 250, a lightweight box that takes up less space than a five-section spiral notebook. Barely tipping the scale at 3.5 pounds, the eSoft appliance is 9 inches wide, 6.25 inches deep, and 1.5 inches tall. The appliance has 256 Mbytes of memory, a built-in 40 Gbyte hard disk drive, and four Fast Ethernet 10/100 ports. The hard disk space is used for in-depth scans of attachments (a virus inside a zip inside a rar file would be detected), quarantines, Bayesian learning systems, and mail queues.

ThreatWall is a little box with big ambitions. Depending on the subscriptions and licenses purchased, the box can more than scanning e-mail traffic to find spam. It can handle URL filtering, web content filtering, malware detection, IM/P2P management, web server/browser protection, and web application protection, using a combination of proxy-based scanning and real-time-packet-based scanning. It also comes integrated with a full mail server that can replace Exchange. Despite all its capabilities, this appliance does not have a firewall and is not intended to sit at the border, but behind an existing network firewall.

The default configuration recommends the ThreatWall be deployed in transparent mode, with the appliance directly connected to the firewall through its WAN port and connecting to a switch and the rest of the network through the LAN port. For this review, reviewers deployed the box as a stand-alone node, connecting the ThreatWall to the switch and acting as just another appliance on the network. The firewall directed mail to the ThreatWall 250, which relayed the filtered traffic to the mail server.

eSoft submitted the ThreatWall 250 with Email ThreatPak, which includes Spamfilter, Gateway AV, Email Content Filter, End User quarantine management, and Botnet Detection. While it's easy to add additional packages through the admin interface, reviewers refrained. Regardless of when the customer adds them, the original solution provider that made the sale gets commission for both the purchase and renewals. It also came with Complete Mail Server, but the appliance was not used as a mail server during the review.

Deployment was fairly straightforward. The appliance was plugged in differently from the default configuration, but a wizard collected about the domains to filter, type of scanning, and actions. The Web-based management interface offers detailed settings options as well as system status and diagnostics. The main dashboard, ThreatMonitor, provides system information such as disk-space used, configured domains, and version numbers for each of the configured protection systems. Email Content Filtering specified keywords to filter on and limits to file attachment sizes. The Spam Filter defined Bayesian filters, custom rules, whitelists, blacklists, and Internet-based resources such real-time blacklists and IP address lookups of known spammers. The Gateway Anti-Virus scanned email messages and Intrusion Prevention could be used to block P2P and IM traffic.

Performance was slightly mixed. At the beginning of the test, the appliance was flagging only 93 percent of the mail traffic as Spam, compared to Symantec's and Sophos' 98 percent. However, there was a key difference in how spam was being flagged. eSoft's product was blocking and rejecting over 93 percent of the traffic while the Sophos appliance had blocked 82 percent. Essentially, ThreatWall was rejecting more mail at the onset while the E1000 was rejecting more mail during the actual scan. The Spam Filter's score thresholds can be tweaked, and a handy table on the ThreatMonitor showed that the average score for High-Spam was 33, and 10.3 for Medium-Spam. Based on this information thresholds were modified for Medium-Spam, which helped reduce the number of spam slipping through and increased the number of flagged spam to 97 percent. As the Bayesian Filter got more fine-tuned and after downloading the SpamFilter Add-In for Outlook 2000-2003 from the management interface, it was easier to flag spam messages.

The wealth of information available in ThreatMonitor was impressive. Reviewers could see at a glance how many messages had been processed, how many were spam, rejected, had viruses, or problems with the contents. A table also explained the reasons why the sessions were rejected. The bulk of the rejected traffic was for unknown recipients, but there were some HELO spoofing and invalid DIA. A separate table also lists incoming user addresses and mail volume, a log of all the mail coming in, types of viruses that were blocked, and IP addresses of blocked clients and servers. A ThreatMap also shows a world map and where the various malware " spam, viruses, spyware, intrusions, and phish " were coming from. This map can also be compared against the world map generated by the eSoft based on global traffic. It was pretty exciting, being able to see how similar the spam traffic entering the Test Center was to global traffic.

eSoft's channel partner program has a category for distributors and a three-tier program for resellers: Gold, Silver and Authorized. While there are no costs associated with joining the program, the partner's level is determined by business volume and number of personnel trained on eSoft's products. Training consists of weekly webinars and quarterly classroom sessions. Training is free other than telecom or travel costs incurred during training.

Pricing for the ThreatWall solution depends on several factors. While the suggested retail price for the appliance is $1,999, the price for the Email ThreatPak can vary depending on subscription length, ranging from $999 for one year to $1,998 for three years. Solution providers can expect margins on the appliance to vary with partner level. Gold partners can expect 30 percent, Silver partners 25 percent, and Authorized with 20 percent. Other revenue opportunities for partners include system monitoring and management services and subscription sales and renewals. Rebates and spiffs are also available for selected programs.

Bottom line

The verdict was mixed at the end of this round of testing. Performance-wise, Symantec's SMS 8360 was up and running with higher catch-rates, learned faster, and ended the two-week test with better results, compared to the eSoft ThreatWall 250 with Email ThreatPak. However, the wealth of information and reporting available to the eSoft appliance blew away anything the reviewers had thus far seen. While each of the previous solutions had good management and reporting capabilities, nothing matched what the eSoft offered. Reviewers joked half-seriously that the perfect solution (so far) would combine Sophos' threat assessment and eSoft's management interface.

Test Center will continue its spam round-up as it works through more appliances, software packages, and SaaS solutions next week. There's no end to spam in sight yet.