The Bug Where Smart Assistants Read a Fact and Still Forget It

The Bug Where Smart Assistants Read a Fact and Still Forget It Source: https://arxiv.org/abs/2606.27472 Paper was published on June 25, 2026 This episode was AI-generated on June 29, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A frontier model can read that you moved to the suburbs and still insist it has no idea where you live — and neither a bigger model nor 24x more memory closes that gap. This paper argues every AI lab that shipped persistent memory in 2026 is treating a behavior problem as a storage problem, and shows the one intervention that actually moves the needle. Key Takeaways: - Why a model can score 92% reading the full conversation but drop to 77% maintaining the same facts from compressed notes — and why that 'supersession gap' is a maintenance problem, not a comprehension one - The 13-to-1 result showing the failure is real and one-directional, not statistical noise - Why a bigger model and 24x more memory both fail to close the gap, with the desk-clutter intuition for why extra storage helps and hurts in equal measure - How a reinforcement-learning reward that targets which version of a fact is *current* nearly doubles held-out accuracy on a small model (9% to 16.7%) - The training curve that 'switches on' exactly when the behavior is learned — the cleanest evidence it's real learning, not luck - Why the headline training result is a single-seed proof of mechanism, and the specific cracks (lenient matching, small question counts, one kind of scale) the episode is honest about 00:00 - A fact it read and lost: The cold open on a model that has the answer in front of it and still says it has no information, and the 15-point gap that frames the episode. 01:45 - Why memory becomes a sticky note: How real systems compress conversations into a notes field, why the agent must actively overwrite stale facts, and the definition of supersession. 04:00 - Is the failure even real?: The clever same-questions experiment comparing full-context to bounded-memory, yielding a 13-to-1 result that isolates maintenance from comprehension. 06:37 - Does a smarter model save you?: Comprehension scales toward solved while the bounded-memory line stalls — proving the skill that scaled isn't the skill that's failing. 07:38 - Can you just buy more memory?: Pulling apart conversation length from compression ratio reveals 24x more memory recovers exactly nothing, even though all answers changed. 11:33 - Training the model to keep facts current: Building a reinforcement-learning reward that rewards temporal currency directly, why synthetic data becomes curriculum, and how GRPO scores without a critic. 16:16 - The curve that switches on: The small model nearly doubles its held-out accuracy, with a training curve that turns on precisely when the behavior is acquired. 18:06 - Where the result actually lands: The honest reservations — single seed, lenient matching, small counts, one kind of scale — and why the diagnosis is strong while the training claim is a first data point. 21:44 - Train the habit or change the substrate?: The bigger reframe that current memory is a learned policy, not a side effect of intelligence, and the closing question posed to listeners. Recommended Reading: - LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory: The benchmark this episode's diagnosis is built on — its knowledge-update questions are exactly what Patel runs under full-context versus bounded-memory conditions. (https://arxiv.org/abs/2410.10813) - DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models: The paper that introduced GRPO, the critic-free reinforcement learning method whose self-terminating batch behavior the episode leans on as evidence of real learning. (https://arxiv.org/abs/2402.03300)