EP100: Meta's Llama 4 Herd Ends Monolithic Models

Meta's Llama 4 announcement introduces a new generation of natively multimodal, open-weight AI models, marking the company's first use of a Mixture-of-Experts (MoE) architecture.The release focuses on two primary models, alongside a preview of a massive third model:Llama 4 Scout: A highly efficient model with 17 billion active parameters (109 billion total) and 16 experts, designed to fit on a single NVIDIA H100 GPU. It features an industry-leading 10-million token context window, which is enabled by a novel "iRoPE" architecture that uses interleaved attention layers without positional embeddings.Llama 4 Maverick: A general-purpose workhorse model with 17 billion active parameters (400 billion total) and 128 experts that fits on a single H100 host. It excels in chat, precise image understanding, and creative writing, outperforming comparable models like GPT-4o and Gemini 2.0 across various benchmarks.Llama 4 Behemoth: A massive 2-trillion total parameter (288 billion active) teacher model. While still in training, it outperforms leading models on STEM benchmarks and was used to distill the capabilities of the smaller Scout and Maverick models.Key Innovations and Training Improvements:Native Multimodality: The models use "early fusion" to seamlessly integrate text and vision tokens during pre-training, allowing them to process and ground up to eight images simultaneously in post-training.Massive Scale: They were trained on a dataset of over 30 trillion tokens spanning 200 languages, which is more than double the size of Llama 3's pre-training mixture.Revamped Post-Training: Meta implemented a new pipeline utilizing lightweight supervised fine-tuning (SFT), online reinforcement learning (RL) with adaptive data filtering, and lightweight direct preference optimization (DPO) to balance intelligence with conversational abilities.Reduced Bias: Meta implemented safeguards that drastically reduce unequal response refusals and bias on contentious political and social topics compared to previous generations.Both Llama 4 Scout and Llama 4 Maverick are currently available for download on platforms like Hugging Face and llama.com.