Deep sequence models tend to memorize geometrically; it is unclear why.

This research introduces the concept of geometric memory to explain how deep sequence models store and reason over atomic facts. Unlike traditional associative memory, which functions as a simple lookup table for co-occurring entities, geometric memory synthesizes global relationships that enable models to solve complex multi-hop reasoning tasks. The authors demonstrate that models can learn to navigate large, unseen graphs by organizing node embeddings into a spatial geometry that reflects the graph's overall structure. Surprisingly, this geometric bias emerges even without specific architectural pressures, capacity limits, or reasoning-based supervision. By comparing Transformers to Node2Vec, the study reveals a spectral bias that naturally directs models toward these powerful, structured representations. Ultimately, these findings challenge the intuition that parametric memory is strictly local, suggesting new ways to improve implicit reasoning and knowledge discovery in language models.