When Turning Experience Into Code Makes Your AI Agent Dumber

When Turning Experience Into Code Makes Your AI Agent Dumber Source: https://arxiv.org/abs/2606.24151 Paper was published on June 23, 2026 This episode was AI-generated on June 24, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. An AI agent that distilled its hard-won experience into reusable code scored ten points worse than an agent with no memory at all. This episode unpacks why the sophisticated-looking move — freezing lessons into callable tools — is also the fragile one, and what the right fix turns out to be. You'll come away understanding the single most basic decision in building agents that learn on the job: when a lesson should stay as soft advice, and when it's earned the right to become code. Key Takeaways: - Why storing an agent's experience as callable code can drop it below an agent with no memory at all — a 22-point collapse the moment it has to generalize - The 'injection asymmetry': text is consumed as adaptable advice you filter through reality, while code is a trusted black box whose flaws propagate to every caller and suppress the agent's own recovery behavior - Metis's 'text first, code earned' policy — sorting experience into plans, facts, and pitfalls, and crystallizing only recurring plans into tools using the desire-path principle - Why the codifier deliberately never reads the messy trajectory, building tools from the clean query pattern instead — and how that lets even failed runs safely count toward codification - The ablation that proves the recurrence gate: an 'Eager' version cost 47% more to build, scored worse, and left over half its tools never invoked - Where the clean story has a seam: the headline result is really about ungated, trajectory-trained, unvalidated code on a single benchmark — not a law that 'code memory is bad' 01:57 - The brilliant employee with amnesia: Frames the core problem: stateless agents lose everything they figure out, and the field hasn't examined how lessons should be stored. 03:01 - Text advice or a black-box tool?: Lays out the fork between storing lessons as adaptable text versus callable code, and why the real difference is how the agent consumes each. 04:50 - The experiment that fixed every variable: Describes the clean diagnostic on AppWorld, splitting executor and reflector models, and measuring construction cost, execution efficiency, and transfer reliability. 08:43 - The 22-point collapse: Reveals the headline reversal: code memory looks great in-sample but collapses 22 points under realistic streaming, dropping below the no-memory baseline. 10:06 - Why the confident tool fails hard: Explains the injection asymmetry through the coworker analogy and why trusted code suppresses an agent's own self-correction. 13:07 - Paving only the paths people walk: Walks through Metis's three design choices — the plans/facts/pitfalls taxonomy, the recurrence gate, and query-only codification — using the desire-path analogy. 18:13 - Does the machinery actually pay off?: Tests the predictions: Metis is more accurate and cheaper at once, and the Eager ablation proves the recurrence gate is a quality filter. 21:41 - The seam in the clean story: The steelman critique: the real claim is about ungated, trajectory-trained code on a single benchmark, with the genuine edge limited to distribution shift. 24:30 - Don't pour the concrete too early: Draws out the durable lesson — store knowledge in a form that follows its properties — and poses the closing question to listeners. Recommended Reading: - ReAct: Synergizing Reasoning and Acting in Language Models: The think-act-observe loop the episode names as the baseline floor every memory variant in Metis is measured against. (https://arxiv.org/abs/2210.03629) - AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents: The exact 457-API simulated benchmark all of the episode's accuracy and token numbers are run on. (https://arxiv.org/abs/2407.18901) - Voyager: An Open-Ended Embodied Agent with Large Language Models: The canonical 'agent builds a reusable skill library of callable code' approach this episode's text-first-code-earned policy is implicitly arguing against. (https://arxiv.org/abs/2305.16291) - Generative Agents: Interactive Simulacra of Human Behavior: A contrasting take where agent experience is stored and retrieved as natural-language memory, the 'soft advice' side of the episode's text-versus-code fork. (https://arxiv.org/abs/2304.03442)