Inheritune: Training Smaller Yet More Attentive Language Models

This research paper investigates the phenomenon of "lazy layers" in large language models (LLMs). Lazy layers occur when deeper layers in LLMs lose the ability to learn meaningful information, leading to a decline in model performance. The authors introduce a new training technique called Inheritune, which addresses this issue by inheriting the initial layers of a larger, pre-trained model and gradually growing the smaller model until it matches or surpasses the performance of the original model. Experiments show that Inheritune effectively trains smaller, high-performing models, demonstrating its potential to make LLM training more efficient and accessible. The paper also analyzes the impact of Inheritune on various model sizes and data regimes, highlighting its efficiency and potential for developing high-quality models even in low-data settings.