Having been a researcher for an AV vendor for nearly a decade, and after having been a part of the Wildlist for a number of years, there came a time when I became very opinionated about security product testing. It’s sort of a hazard of the job of trying to save the world from viruses.
Most of my fellow researchers and I considered passing a certification test proof of basic AV product functionality – a necessary thing, not very exciting. Something like going to the doctor for a yearly checkup: It doesn’t say anything about the four particularly nasty colds we got over the course of the year, and more importantly it doesn’t say much about the improved muscle and stamina we got from taking up a new workout regimen. It’s simply a clean bill of health.
Many of us objected to this existing checkup system, because the malware ecosystem has changed so drastically in the past 5 years, while test methodology had changed very little. In the days of virus outbreaks, you were primarily concerned with the prevalence of samples. When AV products only had “on-demand” and “on-access” scanners, scanning a large number of samples at once was a reasonable test of products’ ability to detect.
Now the malware game is about the lack of prevalence; “keeping your head down,” so you can make money from infections on the sly. So they make thousands of variants to infect those same hundreds-of-thousands of users that would have been hit by the outbreaks of old. AV products have countered this trend by introducing more advanced detection, some of which runs only when an individual file is executed, so it can monitor for suspicious behavior.
Researchers are clamoring for some way to differentiate their new detection strategies, to prove conclusively that they proactively catch more of these thousands of variants before they’re ever even released. Like zero-day detection on steroids. But the basic “clean bill of health” certification testing is not testing for zero-days, and they’re not testing things in run-time. It’s testing the “known universe” – a comprehensive test-bed of all the malware people had seen up to a month or two ago.
So when I heard that West Coast Labs was planning to create some brand new types of tests, I was ecstatic. This is exactly what I’d been hoping to see from the testing industry for years, and I wanted to be a part of it! Our first endeavor is the Real Time test system which tests products against a wide variety of malware from different attack vectors collected in real-time through our global research and honeynet system. Not months-old samples, but hot-off-the-presses malware. And we’re working constantly to expand the types of threat vectors and the samples we’re getting, to assure that what we’re getting is truly representative of the malware world as it is affecting people right now.
This is an incredible opportunity for research organizations within security companies to differentiate themselves. It clearly demonstrates the responsiveness and the proactive detection capabilities of products. Vendors can access this information all day, every day through a secure online interface that gives access to performance data that’s updated every 2 minutes.
All of the researchers I’ve talked to about this have been very excited. In tough economic times like these, it’s even more important to be able to have something to show C-level decision-makers, finance departments and most importantly customers, which gives independent verification of the effectiveness of your work. This is exactly what this test provides – and not a moment too soon!