Why Larger Models Learn More

In this episode: • Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot, challenging the standard sample-efficiency dogma. • The Statistical Bottleneck and Task Utility: Linda explains the paper's synthetic regression setup, detailing how models prioritize tasks based on a utility metric derived from task frequency and complexity. • Gradient Interference and The Update-and-Forget Loop: The hosts dive into the learning dynamics, discussing how frequent tasks overwrite the gradients of rare tasks in small models, and how width solves this interference. • Real-World Validation with OLMo: Linda brings the theoretical findings into the realm of real LLMs, discussing the pre-training of OLMo models and the injection of modular addition and comparison tasks. • Representational Evidence and Final Takeaways: They discuss the mechanistic evidence, including Distributed Alignment Search and Fourier modes, before concluding with practical implications for data curation.