Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away

Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away Source: https://arxiv.org/abs/2605.24517 Paper was published on May 23, 2026 This episode was AI-generated on May 26, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. Standard agent RL throws away 85% of rollouts because the task didn't succeed — but the terminal's responses inside those failed runs contain dense, gradable supervision that nobody was using. A new Microsoft Research paper shows that adding a simple next-token loss on environment outputs roughly doubles task success, recovers most of the value of expensive expert demonstrations, and in some cases lets models improve with no reward signal at all. Key Takeaways: - Why agent RL's reward sparsity is partly an artifact of which tokens we compute loss on, not a property of the task - How ECHO's one-line addition — cross-entropy on terminal output tokens — roughly doubles TerminalBench 2.0 pass rates at 8B and 14B scale - The lambda=0.2 collapse: when the auxiliary weight is too high, models learn to issue boring commands whose outputs are easy to predict - Why ECHO can substitute for the 'interaction prior' half of expert demonstrations but not the 'strategy prior' half - The verifier-free result — improvement with no reward signal on some held-out tasks, and active regression on others — and what that tells us about when prediction-as-learning works - Honest limits: small absolute numbers, untested at higher base capability, and a 'world model' claim that rests on a single transfer experiment 00:00 - The supervision that was already in the rollout: Framing the core observation: failed agent trajectories contain thousands of environment tokens whose gradients GRPO masks out. 03:13 - What ECHO actually changes: The one-line addition of next-token loss on terminal outputs, and the chess-student analogy for why predicting the environment forces understanding. 06:27 - The headline numbers, honestly: Roughly doubled pass rates at 8B and 14B on TerminalBench 2.0 — on a baseline of 2-5%, with timeouts cut in half and faster convergence. 09:41 - Which tokens to predict, and the lambda collapse: Why warning messages had to be excluded, and how setting the auxiliary loss weight too high causes models to game the prediction objective with trivial commands. 12:55 - Substituting for expert demonstrations: ECHO from a raw base model recovers most of the value of 15,000 GLM-4.6 demonstrations — but only the interaction-prior half, not the strategy half. 16:09 - Transfer evidence and the world-modeling claim: ECHO models predict Qwen3-32B's trajectories far better than GRPO baselines, suggesting transferable knowledge of terminal dynamics — though what specifically transferred isn't probed. 15:59 - The verifier-free experiment: Turning off the reward signal entirely and letting environment prediction alone drive improvement — which works on PyTerm, fails on TBLite, and reveals when the method needs action-linked feedback. 22:36 - Steelman, limits, and what to test next: Five honest caveats about the result and the open question of whether ECHO generalizes beyond terminals and beyond low-capability base models. Recommended Reading: - Group Relative Policy Optimization (DeepSeekMath): Introduces the GRPO algorithm that ECHO modifies — essential background for understanding what 'masking out the terminal tokens' actually means in the baseline. (https://arxiv.org/abs/2402.03300) - Curiosity-driven Exploration by Self-supervised Prediction: The canonical prior work on learning from prediction error as an intrinsic signal, which the episode's verifier-free result echoes in a language-model setting. (https://arxiv.org/abs/1705.05363) - Reinforcement Learning with Unsupervised Auxiliary Tasks (UNREAL): A foundational example of adding auxiliary prediction losses to RL agents, useful for contextualizing ECHO against the deeper history of dense-supervision methods the paper doesn't directly compare to. (https://arxiv.org/abs/1611.05397) - SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: Sets the benchmark context for the kind of terminal-agent task ECHO is trying to improve, and frames why doubling a 5% pass rate matters even though the absolute numbers stay small. (https://arxiv.org/abs/2310.06770)