#10 - Qwen2.5-1M Technical Report

This technical report introduces the Qwen2.5-1M series of large language models, significantly enhancing long-context capabilities (up to 1 million tokens) through novel training and inference techniques. Key improvements involve long data synthesis, progressive pre-training, and a multi-stage fine-tuning process. The report also details a newly open-sourced inference framework featuring a length extrapolation method and sparse attention mechanisms to reduce computational costs and improve speed. Evaluations demonstrate substantial performance gains in long-context tasks without sacrificing short-context performance, exceeding even GPT-4o-mini in some areas. The open-source nature of many components promotes broader adoption and development.