While we've recently seen game-changing machine learning breakthroughs in the domains of language, vision, and robotics, it's no secret that security ML progress remains fettered by unverifiable product claims and misleading marketing. In my talk I'll argue that to address this, we need to build a new culture of research transparency in security ML, fostering the same openness that we already bring to subfields like cryptography. Rather than claims of product "secret sauce," we need a culture of publishing our ML models, so they can be openly critiqued. And, instead of making non-reproducible claims about ML model accuracy, we should curate community benchmarks against which we demonstrate the relative efficacy of our ML approaches. In my talk, I'll lay out this argument and introduce the 20 million sample SOREL dataset which my team has released in conjunction with a team at Reversing Labs.