Scaling Laws

Scaling laws describe how language model performance improves with increased model size, training data, and compute. These improvements often follow a power-law, with predictable gains as resources scale up. There are diminishing returns with increased scale. Optimal training involves a balance of model size, data, and compute, and may require training large models on less data, stopping before convergence. To prevent overfitting, the dataset size should increase sublinearly with model size. Scaling laws are relatively independent of model architecture. Current large models are often undertrained, suggesting a need for more balanced resource allocation.