Lessons from Defending the Indefensible

Presented at Black Hat Europe 2015, Unknown date/time (Unknown duration)

For the last year, we've been working hard to optimize CloudFlare's infrastructure to survive different types of denial of service attacks. If you have plenty of servers the usual advice of "buy more bandwidth" may be sufficient, but it certainly wasn't useful to us. At some point you need to do _something_ with the incoming traffic, and the servers have only so many CPU cycles. In this talk, we'll share our experiences in defending our services. We'll go through many layers, from flowspec and sflow, to ethtool tweaks, kernel bypass techniques, iptables examples to useful sysctls. We'll touch on details such as: why increasing backlog queue size may hurt you, why your servers can't send more than 200k syn cookies per second, how to stop a botnet with iptables ipsets and hashlimits, when enabling conntrack makes sense or how to process 10M pps on a single commodity server. Our favorite defense techniques are using BPF, so we will spent a fair bit of time discussing this. We'll discuss what we tried, what worked, what didn't, and why some of the technically sound ideas turned up to be totally impractical. Our experience is in defending HTTP/S and DNS services, on which this talk will focus, but our techniques are applicable to the usual variety of DDoS'es like Chargen, SSDP, NTP or DNS reflection.

Presenters:

  • Marek Majkowski - CloudFlare
    After fruitful encounters with such diverse topics as high performance key value databases, distributed queueing systems, making real time web communication enjoyable, and accelerating the time so that testing servers and protocols takes seconds, Marek Majkowski finally settled for working on DDoS mitigation in the CloudFlare London office, where he appreciates most the parking space for his motorbike.

Links:

Similar Presentations: