Presented at
Black Hat Europe 2020 Virtual,
Dec. 9, 2020, 1:30 p.m.
(30 minutes).
SAST tools are notoriously hard to evaluate and benchmark. The most important thing you want to know about a tool before spending time and money on it: does it give me relevant results? Does it really find the vulnerabilities it promises? Vendors are quick to tell you that their technology will find every vulnerability category out there, and claim to cover every CWE under the sun. But, how do you verify such bold claims? How many vulnerabilities will their tool really uncover, and how many frustrating false positives will you have to trawl through?<br><br>We've all been there: planting mock vulnerabilities in our code bases to challenge a SAST product. It takes a lot of time, and it really only gets you a synthetic set of vulnerabilities to test against. Or you might run tools against one of the many synthetic benchmarking repositories that are riddled with vulnerabilities. Deep inside you know that those codebases have aged and don't really test coverage for modern web frameworks, and rarely test for vulnerabilities that arise due to complex interplay between dependencies and your own code.<br><br>If only we could test tools against *real* vulnerabilities! But hold on… We carefully give every major security vulnerability a globally unique CVE identifier and a collection of metadata. Why not use those! We've triaged hundreds of CVEs in open source codebases and identified the fix commit(s) for every single vulnerability. At Black Hat Europe, we will release this benchmarking dataset and tooling to the open source community.<br><br>This is an initiative by the recently founded Open Source Security Foundation, a part of the Linux Foundation. The working group in which this initiative was developed includes partners from GitHub, Google, Microsoft, Mozilla, and OWASP.
Presenters:
-
Bas van Schaik
- Staff Product Manager for CodeQL, Github
<span>Bas van Schaik is responsible for GitHub's CodeQL security analysis technology. Through his work on CodeQL, he has been involved in the discovery and disclosure of a large number of vulnerabilities in a wide variety of open source projects (including memory corruption in the XNU kernel, and RCEs in Struts and Spring). The CodeQL technology was developed at Semmle, where he was the product lead prior to the acquisition by GitHub in 2019. In a previous life, he completed a PhD at the University of Oxford.</span>
-
Kevin Backhouse
- Security Researcher, GitHub Security Lab
Kevin Backhouse is a member of GitHub Security Lab, where he focuses on finding vulnerabilities in open source projects. He has been doing security research since 2017. His current focus is on helping to improve the security of his favorite operating system - Ubuntu. Prior to that, he spent most of his career working as a compiler engineer. His compiler experience includes working on MATLAB Coder and the ARM C/C++ compiler. More recently, he implemented the range analysis and global value numbering CodeQL libraries for C/C++. He continues to be interested in using program analysis to improve software security.
Links:
Similar Presentations: