EP095: Microsoft Phi-4 Beats Giants With Synthetic Data

The provided paper introduces phi-4, a 14-billion parameter language model developed by Microsoft Research. Unlike typical language models that rely primarily on organic web data, phi-4 achieves state-of-the-art performance for its size by intensely focusing on data quality and strategically integrating synthetic data throughout its entire training process.The model's development is built upon three core pillars:Synthetic Data for Pretraining and Midtraining: The model utilizes high-quality synthetic datasets designed specifically to prioritize structured learning, reasoning, and problem-solving.Organic Data Curation: The researchers meticulously filtered organic data sources—such as web content, licensed books, and code repositories—to extract high-quality "seeds" for their synthetic data pipeline and to include directly in pretraining.Advanced Post-Training Techniques: The paper introduces innovative post-training methods, most notably Pivotal Token Search (PTS) for Direct Preference Optimization (DPO). PTS isolates and optimizes specific tokens that have an outsized impact on a model's probability of generating a correct response, improving its overall reasoning robustness.As a result of these data-centric innovations, phi-4 matches or outperforms much larger foundation models on reasoning-related tasks. Notably, it significantly surpasses its teacher model, GPT-4o, on highly complex benchmarks such as GPQA (graduate-level STEM Q&A) and MATH.