No More Secret Sauce!: How We Can Power Real Security Machine Learning Progress Through Open Algorithms and Benchmarks

Presented at Black Hat Europe 2021, Nov. 11, 2021, 3:20 p.m. (40 minutes)

While we've recently seen game-changing machine learning breakthroughs in the domains of language, vision, and robotics, it's no secret that security ML progress remains fettered by unverifiable product claims and misleading marketing. In my talk I'll argue that to address this, we need to build a new culture of research transparency in security ML, fostering the same openness that we already bring to subfields like cryptography. Rather than claims of product "secret sauce," we need a culture of publishing our ML models, so they can be openly critiqued. And, instead of making non-reproducible claims about ML model accuracy, we should curate community benchmarks against which we demonstrate the relative efficacy of our ML approaches. In my talk, I'll lay out this argument and introduce the 20 million sample SOREL dataset which my team has released in conjunction with a team at Reversing Labs.


Presenters:

  • Joshua Saxe - Chief Scientist, Sophos
    Joshua Saxe is Chief Scientist at Sophos, where he leads artificial intelligence research, development, and operations, and has been involved in shipping more than a dozen machine learning technologies to defend hundreds of millions of endpoints. Before joining Sophos, Joshua spent 5 years leading DARPA funded security data research projects for the US government.

Links:

Similar Presentations: