Introduction to Writing Spiders and Web Agents

Presented at DEF CON 10 (2002), Aug. 4, 2002, noon (50 minutes).

You can have a lot of fun with the Internet by ditching your browser in favor of writing special purpose programs that look for -- or do -- very specific things on the Internet. This session will equip you with techniques to extract and interact with data from web sites without a browser, parse and filter data, follow links, deal with encryption and passwords, and manage terabytes of information. You'll also learn why writing these programs is a useful activity, and walk away with ideas and abilities to write useful spiders or web agents of your own design.

Presenters:

• Michael Schrenk
  Michael Schrenk is a freelance Internet developer, instructor and writer. Much of his consulting business revolves around the creation of spiders, which search the Internet for information of value to his clients. He has also developed web strategies and online applications for Disney, Adidas, Nike and many others.

Links:

• https://defcon.org/html/defcon-10/defcon-10-speakers.html#schrenk
• https://infocon.org/cons/DEF CON/DEF CON 10/DEF CON 10 video/DEF CON 10 - Michael Schrenk - Spiders.mp4
• https://www.youtube.com/watch?v=mb4b-VS2XAg

Similar Presentations:

• Still Hacking Anyway (SHA2017) / Let's hack words: Creative Writing
• AppSec USA 2012 / Why Web Security Is Fundamentally Broken
• BSidesDC 2019 / Hands-on Writing Malware in Go