PodParley PodParley
PodParley
763: Web Scraping + Reverse Engineering APIs

EPISODE · May 1, 2024 · 52 MIN

763: Web Scraping + Reverse Engineering APIs

from Syntax - Tasty Web Development Treats · host Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease. Show Notes 00:00 Welcome to Syntax! 03:13 Brought to you by Sentry.io. 05:00 What is scraping? Socialblade. 08:01 Examples of past scrapers. Canadian Tire. 10:06 Cloud app downloader. 16:13 Other use cases. 16:58 Scraping 101. 17:28 Client Side. 19:08 Private API. Proxyman. 22:40 Server rendered. 23:27 Initial state. 24:57 What format is the data in? Google Puppeteer Extension. 27:08 Working with the DOM. 27:12 Linkedom npm package. 29:02 querySelector everything. 31:28 How to find the elements without classes. 34:08 Use XPath selectors for select by word. 34:53 Make them as flexible as you can. Classes change! 35:10 AI is good at this! 36:26 File downloading. 38:20 Working with protected routes. Proxyman. 40:41 Programatically retrieve authentication keys because they are short-lived. Fetch Cookie. 43:20 Deal-breakers. Mechanical Turk. 44:58 What happened with Amazon? Uniqlo Self-Checkout 46:42 Wes’ portable refrigerator utopia. 47:25 Sick Picks & Shameless Plugs. Sick Picks Scott: KeyboardCleanTool. Wes: Yabai. Shameless Plugs Scott: Syntax on YouTube Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott:X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

NOW PLAYING

763: Web Scraping + Reverse Engineering APIs

0:00 52:32

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

Managing Next Generation Energy Systems Cambridge University Background Stakeholders working with energy systems have to make complex decisions formulated from risk-based assessments about the future. The move towards more renewables in our energy systems complicates matters even further, requiring the development of an integrated power grid and continuous and steady transformation of the UK power system. Network flows must be managed reliably under uncertain demands, uncertain supply, emerging network technologies and possible failures and, further, prices in related markets can be highly volatile. Mathematicians working with engineers and economists, can make significant contributions to address such issues, by helping to develop fit-for-purpose models for next generation energy systems. These interdisciplinary approaches are looking to address a range of associated problems, including modelling, prediction, simulation, control, market and mechanism design and optimisation. This knowledge exchange workshop was part of the four months Res Integrating Nutrition, Psychology and Neuroscience to Measure Infant Development in the UK & Gambia Talk by Dr Sarah Lloyd Fox, Birkbeck College, on infant brain imaging in The Gambia Rise Up Mentor Derek Kempker Welcome to Rise Up Mentor (formerly, Summits of Life), a platform that advocates for lifelong learning and constant improvement in every aspect of life. As someone who’s experienced both the valleys and peaks of life, I understand that while we cannot control all the obstacles that come our way, we do have the power to choose whether we get back up or not. And each time we choose to rise again, we become stronger.But why settle for just getting back up? By consistently striving to become better and wiser, we can reduce our chances of being knocked down in the first place. That’s why I started Rise Up Mentor – to encourage and motivate you to continuously improve in all areas of life, including vocation, marriage, friendship, personal development, health, parenting, lifestyle, spirituality, intellect, and finance.Although I am a Certified Master Life Coach, I don’t consider myself an expert or master, because I don’t feel there is such a thing as mastering life, but it is possible to Kaizen Blueprint Aldo Chandra "Kaizen" is a Japanese term for continuous improvement. This podcast provides a blueprint to learn about health, wealth, relationships and everything else in between. Through our podcast, we strive to inspire, educate, and motivate our audience to cultivate a mindset of lifelong learning, productivity, and personal development. By sharing insights, strategies, and practical tips, we aim to guide listeners on their journey towards realizing their fullest potential, fostering success, and creating lasting positive change.
URL copied to clipboard!