Falcon-H1: Hybrid-Head LLMs for Efficiency and Performance

This source introduces Falcon-H1, a new family of hybrid-head language models designed for efficiency and performance. It explores the architectural innovations, particularly the flexible channel allocation and parallel execution of attention and State Space Model (SSM) components. The document also details various training methodologies, including optimal RoPE base frequency, width-depth trade-offs, and tokenizer improvements, alongside an in-depth analysis of training dynamics such as effective learning rates, weight decay, and the role of µP multipliers. Finally, it outlines the pretraining infrastructure and parallelism strategies like Context Parallelism (CP) and a novel Mixer Parallelism (MP), concluding with extensive multilingual and long-context evaluation results across different model scales.