Scaling Data-Constrained Language Models

Deep dive into Muennighoff et al.'s Scaling Data-Constrained Language Models (2023) — the paper that asks what happens when the balanced scaling recipe demands more fresh, high-quality text than exists. Maya and Leo walk the usable shelf (why the responsibly trainable internet is far smaller than the internet), the second pass (epochs and repetition), and the repetition discount (a few passes are surprisingly close to fresh data before value decays — and excess parameters are discounted too). Then they argue out the workarounds the field is split over: relaxed quality filters, code data, synthetic data and its verification problem, and whether scarcity is even universal once interaction data and retrieval count. Closes Topic 2's arc from predictable scaling through balanced budgets to data economics. Sources: • Scaling Data-Constrained Language Models: https://arxiv.org/pdf/2305.16264 • Training Compute-Optimal Large Language Models: https://arxiv.org/pdf/2203.15556 • Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361