Test-Time Scaling Makes Overtraining Compute-Optimal

Researchers from the University of Wisconsin-Madison and Stanford University propose Train-to-Test (T2) scaling laws to optimize the development and deployment of Large Language Models. Traditional scaling methods like Chinchilla focus primarily on pretraining efficiency, whereas T2 scaling jointly considers model size, training duration, and the compute required for repeated sampling at test-time. The study reveals that when accounting for these inference costs, the most effective strategy shifts toward extreme overtraining, which involves training smaller models on significantly more data than previously recommended. Small, overtrained models often outperform larger counterparts because they allow for more inference samples within the same total compute budget. The authors demonstrate that these T2 scaling predictions remain accurate and beneficial even after models undergo post-training processes like fine-tuning. Ultimately, the work provides a new blueprint for practitioners to maximize performance by balancing training investments with modern test-time scaling strategies.