The Finetuner’s Fallacy: When to Pretrain with Your Finetuning Data

This research introduces specialized pretraining (SPT), a strategy that incorporates domain-specific data directly into the initial pretraining phase rather than reserving it solely for finetuning. By mixing a small percentage of specialized tokens with general web data, models achieve superior performance and faster convergence on niche topics like chemistry, music, and mathematics. This approach effectively addresses the finetuner’s fallacy, proving that early data integration reduces the "tax" of forgetting general knowledge while preventing the overfitting common in standard finetuning. The authors demonstrate that a smaller model using SPT can actually outperform a much larger model trained via traditional methods. Ultimately, the study provides overfitting scaling laws to help practitioners determine the ideal data mixture based on their specific compute budget and dataset size.