Training Compute-Optimal Large Language Models

Deep dive into Hoffmann et al.'s Training Compute-Optimal Large Language Models (2022) — the Chinchilla paper that re-measured the parameters-versus-tokens trade-off and found a generation of large models undertrained. Maya and Leo walk the three landmarks: the rebalance — under a fixed compute budget, model size and training tokens should scale roughly together; the rematch — a seventy-billion-parameter model trained on far more data outperforming much larger models like Gopher at a comparable budget, while also being far cheaper to serve; and the fine print — tokens are not interchangeable, loss is not a task evaluation, and the balanced recipe points straight at a data bottleneck. They argue out whether frontier labs are still right to train past the optimum, and set up the data-constrained regime next episode. Sources: • Training Compute-Optimal Large Language Models: https://arxiv.org/pdf/2203.15556 • Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361