Differential Transformer

The research paper introduces the Differential Transformer, a new architecture for large language models that aims to improve the performance of these models by reducing the amount of attention they pay to irrelevant information. This architecture accomplishes this through a differential attention mechanism that calculates attention scores as the difference between two separate attention maps. This process effectively cancels out noise in the attention scores, encouraging the model to focus on more relevant information. The paper highlights the potential benefits of this architecture through various experiments, showcasing its superior performance in tasks like long-context modeling, key information retrieval, and in-context learning, while also mitigating issues like hallucination and activation outliers.