Optimized Web Crawling

from Linear Digressions · host Katie Malone

Got a fun optimization problem for you this week! It’s a two-for-one: how do you optimize the web crawling logic of an operation like Google search so that the results are, on average, as up-to-date as possible, and how do you optimize your solution of choice so that it’s maintainable by software engineers in a huge distributed system? We’re following an excellent post from the Unofficial Google Data Science blog going through this problem. Relevant links: http://www.unofficialgoogledatascience.com/2018/07/by-bill-richoux-critical-decisions-are.html

Episode metadata supplied by the publisher feed · Published Oct 28, 2018

Embed this episode

Attribution link and audio player

Ready to play

Optimized Web Crawling

0:00 21:32

1×

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

The Talk Show With John Gruber Daring Fireball / John Gruber The director’s commentary track for Daring Fireball. Long digressions on Apple, technology, design, movies, and more. Getting Into Infosec Ayman Elsawah Interviews with people who have transitioned and got jobs in #infosec and #cybersecurity so you can learn and be inspired from their experience. There is no linear path into the field of Information Security, so the hope is that you will resonate with at least one of the guests. Some of my guests were teachers, paralegals, librarians, military vets, developers, and IT help desk techs (to name a few) before transitioning. Also featuring "spoof" ads poking fun at the industry.

Frequently Asked Questions

How long is this episode of Linear Digressions?

This episode is 21 minutes long.

When was this Linear Digressions episode published?

This episode was published on October 28, 2018.

Can I download this Linear Digressions episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.

URL copied to clipboard!