Self-Distillation for Data-Scarce Language Model Pretraining

This research paper investigates self-distillation as a powerful regularization technique for pretraining language models when high-quality data is in short supply. By comparing various training strategies across different model scales and data scarcity levels, the authors demonstrate that self-distillation significantly outperforms both direct training and standard methods like weight decay or exponential moving averages. The study identifies a specific crossover threshold where distillation becomes superior, particularly when the available data is less than one-fourth of the amount prescribed by Chinchilla scaling laws. Practical results suggest that using larger models with natural teacher temperatures provides the most effective supervision, preventing the rapid overfitting typically seen in data-constrained environments. Ultimately, the work advocates for self-distillation as a robust alternative for improving model performance when compute resources outpace the available data pool.