Scaling and Training Large Models Efficiently

Topic 2 opens with the question the Transformer made urgent: once you can build big, how should one fixed training budget be split between model size, training tokens, and data quality? Maya and Leo stage the scale-first versus compute-optimal argument in its strongest forms, introduce the smooth-curve predictability of scaling laws, the four interacting knobs of scale, and the two-bills view of training versus inference cost — then map the three deep-dives: Kaplan's scaling laws, Chinchilla's budget correction, and the data-constrained regime where fresh text runs short. Sources: • Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361 • Training Compute-Optimal Large Language Models: https://arxiv.org/pdf/2203.15556 • Scaling Data-Constrained Language Models: https://arxiv.org/pdf/2305.16264