When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model

When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model Source: https://arxiv.org/abs/2606.06324 Paper was published on June 04, 2026 This episode was AI-generated on June 5, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. An AI agent confidently reports a task complete while the database shows nothing actually happened — and no prompt edit on earth can fix it. A new paper argues that for a huge share of agent failures, the model is already good enough, and the real bug lives in the deterministic scaffolding around it. The payoff: that scaffolding is just software, which means you can actually diagnose and repair it. Key Takeaways: - Why many agent failures are 'silent successes' — the harness marks a task complete even though nothing changed in the world — and why benchmark scores actively hide them - How HarnessFix borrows a compiler trick (a normalized intermediate representation) to turn messy, framework-specific traces into something you can analyze uniformly - The four-stage pipeline — abstraction, diagnosis, repair, validation — and why repairs draw from a fixed, vetted catalog distilled from real repo fixes rather than letting the agent rewrite itself freely - Why a prompt-only version of the system gets zero improvement on Terminal-Bench while the full system fixes lifecycle, observability, and verification flaws prompts can't reach - The honest limitations: the system largely grades its own diagnoses, raw gains are small (six tasks to nine), results are single-run on one model, and the 'beats human harnesses' comparison isn't a clean head-to-head 08:04 - The bill-splitting disaster: An agent sends Venmo requests, reports success, and yet zero payments exist — the opening example that lands the paper's whole thesis. 03:22 - Reframing the agent as model plus harness: Why the deterministic software wrapping the model — the harness — is the real culprit, and why that matters: it can be debugged. 06:44 - Taming the trace: How agent traces are a rambly mess with no common format, and how a compiler-style intermediate representation normalizes them and tags each step's role, success, and world-effect. 10:07 - The four-stage repair pipeline: Walking through abstraction, diagnosis, recurring flaw records, code patches, and validation as a named assembly line. 13:29 - Repair by catalog, not by improvisation: The opinionated design choice to fix flaws from a fixed menu of vetted operators — the surgeon analogy — and the regression-aware acceptance bar that gates every patch. 16:52 - Watching the pipeline fix the bill-splitter: How the system diagnoses three stacked harness flaws, consolidates them, and produces a completion guard no prompt edit could have delivered. 20:14 - The numbers and the prompt-only ablation: Held-out improvements of fifteen to fifty percent across four benchmarks, beating hand-built human harnesses, with concrete patches like blocking session-killing commands. 23:37 - Taking the knife to it: The critiques: the system largely grades its own diagnoses, the catalog can't reach novel flaws, gains are small and single-run on one model, and what the regression-guard ablation reveals about the design. Recommended Reading: - ReAct: Synergizing Reasoning and Acting in Language Models: Defines the reasoning-and-acting loop that constitutes the core of the 'harness' this episode dissects, giving listeners the baseline agent architecture HarnessFix repairs. (https://arxiv.org/abs/2210.03629) - Reflexion: Language Agents with Verbal Reinforcement Learning: A leading example of the trace-driven self-improvement methods the episode contrasts against, since Reflexion edits the model's prompt/memory rather than the runtime scaffolding. (https://arxiv.org/abs/2303.11366) - Voyager: An Open-Ended Embodied Agent with Large Language Models: Represents the self-modifying-agent lineage the paper deliberately rejects, letting listeners weigh free self-rewriting against HarnessFix's constrained repair-operator catalog. (https://arxiv.org/abs/2305.16291) - SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: Provides the repository bug-fixing benchmark style underlying one of the four evaluation domains, illustrating the kind of pass/fail scoring the episode critiques for hiding silent-success failures. (https://arxiv.org/abs/2310.06770)