Why LLM Progress is Getting Harder

Jed Sundwall and Drew Breunig explore why LLM progress is getting harder by examining the foundational data products that powered AI breakthroughs. They discuss how we've consumed the "low-hanging fruit" of internet data and graphics innovations, and what this means for the future of AI development.The conversation traces three datasets that shaped AI: MNIST (1994), the handwritten digits dataset that became machine learning's "Hello World"; ImageNet (2008), Fei-Fei Li's image dataset that launched deep learning through AlexNet's 2012 breakthrough; and Common Crawl (2007), Gil Elbaz's web crawling project that fueled 60% of GPT-3's training data. Drew argues that great data products create ecosystems around themselves, using the Enron email dataset as an example of how a single data release can generate thousands of research papers and enable countless startups. The episode concludes with a discussion of benchmarks as modern data products and the challenge of creating sustainable data infrastructure for the next generation of AI systems.Links and Resources:- Common Crawl Foundation Event - October 22nd event at Stanford!- Cloud-Native Geospatial Forum Conference 2026 - 6-9 October 2026 at Snowbird in Utah!- Why LLM Advancements Have Slowed: The Low-Hanging Fruit Has Been Eaten - Drew's blog post that inspired this conversation- Unicorns, Show Ponies, and Gazelles - Jed's vision for sustainable data organizations- ARC AGI Benchmark - François Chollet's reasoning benchmark- Thinking Machines Lab - Mira Murati's reproducibility research lab- Terminal Bench - Stanford's coding agent evaluation benchmark- Data Science at the Singularity - David Donoho's masterful paper examining the power of frictionless reproducibility- Rethinking Dataset Discovery with DataScout - New paper examining dataset discovery- MNIST Dataset - The foundational machine learning dataset on Hugging FaceKey Takeaways1. Great data products create ecosystems - They don't just provide data, they enable entire communities and industries to flourish2. Benchmarks are data products with intent - They encode values and shape the direction of AI development3. We've consumed the easy wins - The internet and graphics innovations that powered early AI breakthroughs are largely exhausted4. The future is specialized - Progress will come from domain-specific datasets, benchmarks, and applications rather than general models5. Data markets need new models - Traditional approaches to data sharing may not work in the AI era