Bridging Compute- and Data-Optimal Pretraining

In this episode: • The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling. • The Compute-Data Scaling Law: Linda introduces the new CD-scaling law, focusing on the token effectiveness function that bridges compute-optimal and data-optimal scaling. • Eta and the Saturation Ceiling: A deep dive into how data effectiveness decays with model size and data availability, culminating in the R-star saturation ceiling. • Paraphrasing vs Repetition: Discussion on practical data expansion strategies, revealing why simple multi-epoch repetition surprisingly beats paraphrasing for large models. • Three Operational Regimes: Summarizing the compute-bound, data-bound, and model-bound regimes, providing a map for future LLM pretraining resource allocation.