EP073: Mixtral 8x7B Sparse Experts Beat Giants

The paper introduces Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model developed by Mistral AI and released under the open-source Apache 2.0 license.Key Highlights:Architecture and Efficiency: Mixtral 8x7B uses a decoder-only transformer architecture where each layer consists of 8 distinct feedforward blocks, or "experts". For every token at each layer, a routing network selects the top two experts to process the token and additively combine their outputs. Consequently, while the model has a total of 47 billion parameters, it only uses 13 billion active parameters per token. This sparse activation provides faster inference speeds at low batch sizes and higher throughput at large batch sizes compared to dense models.Performance: Pretrained with a context window of 32k tokens, the model matches or outperforms larger models like Llama 2 70B and GPT-3.5 across a wide variety of benchmarks. It demonstrates vastly superior capabilities in mathematics, code generation, and multilingual tasks (including French, German, Spanish, and Italian) while using 5x fewer active parameters than Llama 2 70B.Instruct Model: The authors also released Mixtral 8x7B – Instruct, a model fine-tuned using supervised fine-tuning and Direct Preference Optimization (DPO). This chat model surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B on human evaluation benchmarks and exhibits reduced social biases.Routing Behavior Analysis: An analysis of the model's routing mechanism revealed that experts do not strongly specialize in specific topics (like biology or philosophy). Instead, they exhibit structured syntactic behavior—such as routing specific syntax structures like Python indentations to the same experts—and show high temporal locality, meaning consecutive tokens are often assigned to the same experts.