“Well, robots.txt is just a handshake agreement that…” — The Vergecast at 23:43
“Well, robots.txt is just a handshake agreement that everyone made. Like, David, you wrote a piece about this almost a year ago now. Yeah. Like, this was a very informal, like, how do we reduce the cost of all the bots showing up on our website? Because it was really expensive back then. And now, like, the world…”
0:00
1:48:49