Michael Schrenk - Screen Scraper Tricks Extracting Data from Difficult Websites - Video and Slides
from DEFCON 17 [Video and Slides] Speeches from the Hacker Convention. Hosted by DEF CON Announcements
Screen Scraper Tricks: Extracting Data from Difficult Websites Michael Schrenk Screen scrapers and data mining bots often encounter problems when extracting data from modern websites. Obstacles like AJAX discourage many bot writers from completing screen scraping projects. The good news is that you can overcome most challenges if you learn a few tricks. This session describes the (sometimes mind numbing) roadblocks that can come between you and your ability to apply a screen scraper to a website. You'll discover simple techniques for extracting data from websites that freely employ DHTML, AJAX, complex cookie management as well as other techniques. Additionally, you will also learn how "agencies" create large scale CAPTCHA solutions. All the tools discussed in this talk are available for free, offer complete customization and run on multiple platforms. Michael Schrenk is a webbot developer and the author of "Webbots, Spiders, and Screen Scrapers" (2007, No Starch Press). He has also written for ComputerWorld, php|architect and Web Techniques magazines. Mike also gave presentations at DEFCON X, XI and XV. He works for a wide range of clients across North American as well as in Russia, Spain and The Netherlands. Stop by www.schrenk.com and say hello. return to top
Embed this episode
No transcript yet.