Bots Are Just Another Customer - Dan Kurtz, Miss Pepper AI

Traditional Googlebot crawling is falling while AI retrieval bots climb. Dan Kurtz of Miss Pepper AI reads it off his own server logs, and his conclusion is not that SEO is over. It is that we have done this before. We have been multi-purposing content into infographics, reels and press releases for twenty years because humans have preferred content types. Bots have one too. Key takeaways Bots are a customer segment, not a technical problem. They have a format preference, and serving it is ordinary multi-purposing. The downside of publishing a machine-readable companion is bounded. Worst case, the crawler ignores it. Linking to the file is the mechanism. A companion file nobody links to is a file nobody fetches. Deliverable-count retainers are a symptom. Hitting a monthly quota is not the same as paying attention. An SSL certificate reissue may force AI recrawls. Offered as an untested hypothesis, with the caveat attached. On this page The argument: bots as a customer segment What the crawl data actually shows The agentic flywheel Why he refuses deliverable-based retainers Chapters and timestamps People, tools and sources mentioned Questions this episode answers Go deeper The argument: bots as a customer segment As SEOs and content marketers, we’ve been doing that for decades. We call it multi-purposing. Bots have a preferred content type. Technically, they’re just an additional customer. — Dan Kurtz, 37:42 The practical version is a plain-text or markdown file sitting alongside the HTML page. Dan’s case for doing it is that the risk is asymmetric: I would absolutely spend the time to go ahead and do that and invest in it, because the worst thing that can happen is they pass it. They fly over it. — Dan Kurtz, 37:42 Then the step almost everyone skips. “That’s just a matter of directing them to it next time they come to the website and saying, hey, this file exists.” We took that literally and published machine-readable companions across twelve episodes, linked from every post. One thing we found by accident: our own flagship site had been returning 429 to GPTBot on every page. Not everyone agrees. Malte Landwehr makes the opposite case, that a second URL carrying the same content is not cost-free, and would serve markdown by user agent at the same URL instead. Two practitioners, two defensible positions, which is usually the sign that the question is open rather than solved....