763: Web Scraping + Reverse Engineering APIs - Syntax

What this episode covers

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease. Show Notes 00:00 Welcome to Syntax! 03:13 Brought to you by Sentry.io. 05:00 What is scraping? Socialblade. 08:01 Examples of past scrapers. Canadian Tire. 10:06 Cloud app downloader. 16:13 Other use cases. 16:58 Scraping 101. 17:28 Client Side. 19:08 Private API. Proxyman. 22:40 Server rendered. 23:27 Initial state. 24:57 What format is the data in? Google Puppeteer Extension. 27:08 Working with the DOM. 27:12 Linkedom npm package. 29:02 querySelector everything. 31:28 How to find the elements without classes. 34:08 Use XPath selectors for select by word. 34:53 Make them as flexible as you can. Classes change! 35:10 AI is good at this! 36:26 File downloading. 38:20 Working with protected routes. Proxyman. 40:41 Programatically retrieve authentication keys because they are short-lived. Fetch Cookie. 43:20 Deal-breakers. Mechanical Turk. 44:58 What happened with Amazon? Uniqlo Self-Checkout 46:42 Wes’ portable refrigerator utopia. 47:25 Sick Picks & Shameless Plugs. Sick Picks Scott: KeyboardCleanTool. Wes: Yabai. Shameless Plugs Scott: Syntax on YouTube Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott:X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

Kaizen Blueprint Aldo Chandra "Kaizen" is a Japanese term for continuous improvement. This podcast provides a blueprint to learn about health, wealth, relationships and everything else in between. Through our podcast, we strive to inspire, educate, and motivate our audience to cultivate a mindset of lifelong learning, productivity, and personal development. By sharing insights, strategies, and practical tips, we aim to guide listeners on their journey towards realizing their fullest potential, fostering success, and creating lasting positive change. Chewing the Fat with WorkForge WorkForge Bite-Sized Conversations for Building a Stronger Workforce Welcome to Chewing the Fat, a podcast delving deep into the world of food manufacturing. Dive into real conversations around critical topics like staffing, retention, onboarding, and career development in this essential industry. Subscribe now to gain insights from your peers, subject matter experts and more on the biggest issues facing food manufacturers today: -Hiring and retaining employees -Addressing the challenges of the Silver Tsunami -Improving time to productivity of new employees -Engaging employees from hire to retire And more... Tune in to Chewing the Fat, a WorkForge podcast, and join the conversation on how to build and sustain a resilient, high-performing workforce in food manufacturing. Darknet Discussions Darknet Discussions Welcome to "Darknet Discussions," the podcast that gets into the shadows of the internet to bring you the most intriguing, enlightening, and sometimes unsettling stories from the dark web. Hosted by seasoned darknet aficionados, each episode of "Darknet Discussions" explores the intricate dynamics of darknet markets, cybersecurity threats, and the digital underworld. Join us as we interview experts, discuss the latest trends in cybercrime, and shed light on the technologies that operate beneath the surface of everyday internet use. Also, we occasionally go off on a tangent about something completely unrelated. The Protocol CoinDesk Dive deep into the blockchain realm with The Protocol Podcast, where we unravel the intricate technologies powering cryptocurrencies like Bitcoin and Ethereum. Join us on a journey through the labyrinthine layers of blockchain innovation, as tech-savvy developers sculpt the future of finance and the decentralized web. Led by CoinDesk's adept journalists, we dissect the freshest news and project revelations, demystifying the mechanics and significance of it all for those hungry to grasp the inner workings of this dynamic and rapidly evolving industry.Meet your hosts: Brad Keoun, Sam Kessler, and Margaux Nijkerk…and tune in, techies!

Frequently Asked Questions

How long is this episode of Syntax - Tasty Web Development Treats?

This episode is 52 minutes long.

When was this Syntax - Tasty Web Development Treats episode published?

This episode was published on May 1, 2024.

What is this episode about?

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and...

Can I download this Syntax - Tasty Web Development Treats episode?

Yes, you can download this episode by clicking the download button on the episode player, or subscribe to the podcast in your preferred podcast app for automatic downloads.