Miras: Sculpting Memory and Retention in Sequence Architectures

The text outlines the Miras framework, a generalized structure for designing efficient sequence models inspired by human associative memory and cognitive processes. Miras unifies and categorizes neural architectures, including Transformers and recurrent neural networks, based on four critical design choices: the memory architecture, the attentional bias objective, the retention gate, and the memory learning algorithm. The authors observe that standard models often rely on limited objectives and propose alternative biases, such as $l_p$ losses and Huber loss, and sophisticated retention gates rooted in concepts like Bregman and KL divergence, to improve memory management. This new perspective leads to the introduction of three novel sequence models—Moneta, Yaad, and Memora—which are shown to outperform existing state-of-the-art models in language modeling and recall-intensive benchmarks. Ultimately, the framework provides guidelines for developing a new generation of expressive and robust architectures capable of handling long contexts.