Scaling Laws for Neural Language Models

Deep dive into Kaplan et al.'s Scaling Laws for Neural Language Models (2020), the paper that made giant training runs forecastable. Maya and Leo walk the three landmarks: the ruler — loss falls along smooth power laws in parameters, data, and compute, so cheap pilot runs predict frontier runs; the early exit — larger models learn more per token, so a fixed budget should buy a huge model trained on modest data and stopped before convergence; and the edge of the map — loss is a proxy, curves are fitted to a measured range, and averages can hide brittle rare-task behavior. They stage the real argument between curve-trusting planners and loss-as-proxy skeptics, and set up Chinchilla's revision next episode. Sources: • Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361 • Training Compute-Optimal Large Language Models: https://arxiv.org/pdf/2203.15556