Differential Transformer

🎧 Differential TransformerThe paper introduces the Differential Transformer, a new architecture for large language models (LLMs) that aims to improve their ability to focus on relevant information within long sequences. It achieves this by introducing a differential attention mechanism which calculates attention scores as the difference between two separate softmax attention maps, effectively canceling out noise and promoting sparse attention patterns. This enhanced focus on relevant context leads to improvements in various tasks, including long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reducing activation outliers. The paper provides experimental evidence to support these claims, showcasing the Differential Transformer's superiority over traditional Transformers in several scenarios.📎 Link to paper