Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors

In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can:Test selectors before writing spidersInspect HTML responses instantlyExperiment with scraping logic🔹 Key Objects Inside Shellresponse → HTML content of the pagerequest → HTTP request detailsspider → scraper context👉 Key InsightYou can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports:🌐 Live websites📄 Local HTML files👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key InsightCSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell:Works on cached HTMLWon’t reflect live changes unless restarted👉 Key InsightAlways restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/pStarts from rootVery strict🔹 Relative Path//div/pSearches anywhereMore flexible7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key InsightXPath is powerful for uncertain or messy HTML structures9. CSS vs XPathFeatureCSSXPathSimplicity✔️ Easy❌ More complexPower❌ Limited✔️ Very powerfulFlexibilityMediumVery high10. Mental ModelThink of Scrapy Shell as:A laboratoryCSS = quick filtersXPath = surgical precision toolsFinal TakeawayScrapy Shell bridges the gap between:👉 “guessing selectors”and👉 “engineered extraction logic”Once you master CSS + XPath inside the shell, you can confidently build spiders that work on even the most complex websites.You can listen and download our episodes for free on more than 10 different platforms:https://linktr.ee/cybercode_academy