Presented at
VB2016,
Oct. 5, 2016, 2:30 p.m.
(30 minutes).
Typical anti-virus (AV) evaluation methods are based on automated tests performed in controlled environments. While these tests are adequate to evaluate the efficacy of AV products under specific scenarios, they do not measure the field efficacy of AV products as deployed on machines operated by real users. In addition, they are limited in determining how AV performance varies as a function of user action, machine configuration, and type of threat.
To address these limitations, we develop a novel methodology to conduct AV tests based on real-life usage. In such tests, AV products are evaluated though long-term field studies where actual customers use the products in environments of their choice. In this paper, we describe our approach and present the results of a first test of this kind, aimed at evaluating AV products under real-life scenarios rather than in controlled environments.
Using data collected from *Microsoft Windows Malicious Software Removal Tool* (*MSRT*) and* Microsoft Windows Defender* on billions of machines, we conducted a large-scale comparative test of AV products. We describe our experimental design and present the comparative results obtained. Interestingly, we notice important differences in the rankings depending on the populations considered. In other words, the performance of AV varies significantly as a function of user factors such as age group, gender, and socio-economic status. Finally, we discuss the benefits and limitations of this type of AV test and highlight how they are complementary to other traditional AV evaluation methods.
Presenters:
-
Dennis Batchelder
- AppEsteem, and formerly Microsoft
Dennis Batchelder Dennis Batchelder is the President of AppEsteem Corporation, where he's formulating better approaches to eradicating unwanted software. He spent eight years at Microsoft, where he led the company's anti-malware efforts to protect billions of customers through real-time anti-malware products and services, industry partnerships, and continuous analysis of threat intelligence using machine learning and the cloud. Prior to Microsoft, Dennis owned the threat and security information management product lines as a Senior Vice President at Computer Associates, which he joined after founding, running, and selling them a network security product company. Dennis has worked for more than 20 years in the security industry holding various leadership roles in the US and India. He lives in Seattle, Washington. Dennis is the author of the Soul Identity series of techno-thriller novels. @denbatch
-
Fanny Lalonde Lévesque
- École Polytechnique de Montréal
Fanny Lalonde Lévesque Fanny Lalonde Lévesque is a Ph.D. student in the Department of Computer & Software Engineering at the École Polytechnique de Montréal. She received her B.Eng. in software engineering (2010) and her Master of Applied Sciences (2013) degrees from the École Polytechnique de Montréal. Her main research interests lie in security product testing methodologies and in identifying and understanding human and technological risk factors leading to malware infections.
-
Glaucia Young
- Microsoft
Glaucia Young Glaucia Young is an engineering manager at Microsoft Malware Protection Center (MMPC). Since 2011, Glaucia has been leading MMPC's efforts to help protect billions of computers from malware through delivery of high quality anti-malware products and services, using innovative testing strategies and data-driven engineering. Glaucia has worked for more than 18 years in the technology industry, holding various roles in testing/QA, software engineering, data science, and management in the US. A native of Brazil, she currently lives in Cle Elum, Washington.
-
Jose M. Fernandez
- École Polytechnique de Montréal
Links:
Similar Presentations: