After Titans: Behrouz on Nested Learning and Hope

Episode title: After Titans: Behrouz on Nested Learning and Hope The followup to our Titans episode, by the same core team a year later. Behrouz, Razaviyayn, Zhong, and Mirrokni (Google Research) generalize the Titans bet — that long-term memory should be a learnable module updated at test time — into a broader paradigm they call Nested Learning, where a "deep" architecture is really a hierarchy of nested optimization problems each compressing its own context flow. The episode walks through their three core contributions: (1) reframing standard optimizers like Adam and SGD-with-Momentum as associative-memory modules that compress gradient information, then proposing more expressive optimizers with their own deep memory; (2) a self-modifying sequence model whose update rule is itself learned end-to-end — the natural generalization of the test-time-learnable memory module Titans introduced; (3) a continuum memory system that replaces the traditional short-term-vs-long-term dichotomy with a continuum across multiple update rates. Combining the self-modifying module with the continuum memory system produces Hope, a continual-learning architecture reported promising on language modeling, knowledge incorporation, few-shot generalization, continual learning, and long-context reasoning. The hosts treat Hope as the next concrete instance of the Titans → Nested Learning research arc, stress-test the novelty against established meta-learning and fast-weights literature, and distinguish what the paper actually shows from what its framing suggests. Sources: Nested Learning: The Illusion of Deep Learning Architectures — Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni (Google Research, NeurIPS 2025) — arXiv 2512.24695 https://arxiv.org/pdf/2512.24695 Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni (Jan 2025) — arXiv 2501.00663 https://arxiv.org/pdf/2501.00663 AI Post Transformers — "Titans: Learning to Memorize at Test Time" https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp3 Interactive Visualization: After Titans: Behrouz on Nested Learning and Hope