Titans: Long-Term Neural Memory for Scaling Context

The provided text details the development of Titans, a new family of neural architectures designed to overcome the fixed context window and computational scaling issues inherent in modern Transformers. The core of this system is a novel neural long-term memory module that operates as a meta in-context learner, dynamically updating its weights based on a calculated "surprise" metric that includes both momentum and an adaptive forgetting mechanism. This mechanism allows the module to store persistent, historical information, thereby acting as long-term memory while the attention component handles immediate context as short-term memory. The paper presents three distinct architectural ways to integrate this new component: Memory as a Context (MAC), Memory as a Gate (MAG), and Memory as a Layer (MAL). Empirical testing across diverse tasks, including language modeling and extreme-length reasoning benchmarks, demonstrates that Titans achieve superior performance and efficiency compared to baseline models, successfully scaling to context window sizes exceeding two million tokens.