Stronger Models are NOT Stronger Teachers for Instruction Tuning

This research paper investigates the impact of different language models (LLMs) used as "teachers" to generate synthetic responses for instruction tuning. The authors demonstrate a surprising phenomenon they call the "Larger Models' Paradox," where larger and supposedly "stronger" teacher models do not always lead to improved instruction-following abilities in smaller base models. They propose a novel metric called Compatibility-Adjusted Reward (CAR) to better predict the effectiveness of teacher models, taking into account the compatibility between the teacher and the base model being fine-tuned. The study challenges the common assumption that larger LLMs are always better teachers and suggests that a more nuanced understanding of compatibility is needed for successful instruction tuning.