When Agent Benchmarks Lie: The Harness Problem in Open-Source AI

When Agent Benchmarks Lie: The Harness Problem in Open-Source AI Source: https://arxiv.org/abs/2605.15040 Paper was published on May 14, 2026 This episode was AI-generated on May 15, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A software-engineering agent scores 62% on its native test setup and 3.6% when you swap the wrapper around it. A new paper called Orchard argues this isn't a bug in one system — it's an indictment of how the entire open-source agent field has been measuring progress, and it offers an infrastructure-first fix that costs ten times less and actually generalizes. Key Takeaways: - Why most reported agent benchmark scores measure harness-fit rather than underlying capability, and how a cross-harness test exposes the gap - How treating the sandbox layer as a thin, generic service (rather than baked-in plumbing) cuts training costs roughly 10x versus managed services like E2B and Daytona - Credit-assignment SFT: extracting partial supervision from failed teacher trajectories by finding the rising segment before the critical mistake - Balanced Adaptive Rollout (BAR): a self-pacing RL technique that stops generating rollouts once a prompt yields a useful mix of wins and losses - The surprising GUI result — a 4B-parameter student beating its 235B teacher — and why environment-grounded RL teaches something distillation can't - Honest limitations the paper undersells: the cross-harness comparison is partly confounded, RL gains are measured on a curated subset, and the whole recipe depends on a few open frontier teacher models staying open 00:00 - The 62-to-3.6 collapse: Why swapping the harness around the same model weights can demolish benchmark scores, and what that says about the field. 03:27 - Agent, harness, and what Orchard actually is: Vocabulary for ReAct loops and harnesses, plus the dual nature of Orchard as both infrastructure and training recipes. 06:54 - Sandboxes as a thin service: The architectural case for pulling the environment layer out behind a small REST API, and the cost and latency numbers that follow. 10:21 - Credit-assignment SFT: salvaging failed trajectories: Using hindsight from the teacher model to extract training signal from the productive prefix of attempts that ultimately failed. 13:49 - Balanced Adaptive Rollout for RL: A self-pacing rollout strategy that ensures every gradient batch contains both successes and failures, turning prompt difficulty into a runtime curriculum. 23:17 - The cross-harness experiment: Evaluating competing agents on harnesses they weren't trained against, and what the resulting collapses reveal about generalization. 20:43 - When a 4B student beats a 235B teacher: The browser-agent result, the tennis-coach analogy for why outcome-grounded RL can exceed imitation, and the captcha problem hiding in the training data. 24:11 - What the paper undersells: Honest critiques: confounded harness diversity, RL gains measured on a curated subset, and the ecosystem's fragile dependence on a few open teacher models. Recommended Reading: - SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: The benchmark at the center of the episode's harness-collapse story — worth reading to understand what '67.5 percent' actually measures and why the harness wraps around it. (https://arxiv.org/abs/2310.06770) - ReAct: Synergizing Reasoning and Acting in Language Models: The reason-then-act loop that Bella defines early on as the 'body the model lives inside' — foundational for understanding what a harness even is. (https://arxiv.org/abs/2210.03629) - DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO): Introduces GRPO, the rollout-comparison RL algorithm that Eric walks through before explaining why Balanced Adaptive Rollout exists to fix its all-success/all-failure waste problem. (https://arxiv.org/abs/2402.03300) - SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering: A direct prior argument that the agent-computer interface — what this episode calls the harness — is itself a first-class design variable, not just plumbing around the model. (https://arxiv.org/abs/2405.15793)