DeepSeek Engram: Conditional Memory via Scalable Lookup
from The Gist Talk Hosted by kw
This episode introduces Engram, a new architectural module that integrates conditional memory into Large Language Models to handle static knowledge more efficiently. Traditional models often waste computational depth simulating memory retrieval, but Engram uses $N$-gram lookup tables to retrieve information in constant time. By balancing this memory module with Mixture-of-Experts (MoE) computation, the authors discovered a U-shaped scaling law that optimizes performance for a fixed parameter budget. Experimental results show that Engram-enhanced models significantly outperform standard MoE baselines in general reasoning, coding, and long-context tasks. Mechanistically, the module functions by offloading local pattern reconstruction from early layers, effectively increasing the model's functional depth. Furthermore, its deterministic retrieval allows for efficient host memory offloading, enabling massive parameter scaling with minimal impact on inference speed
Embed this episode
No transcript yet.