The Agent Failed — But Did the Instructions Deserve to Be Followed?

The Agent Failed — But Did the Instructions Deserve to Be Followed? Source: https://arxiv.org/abs/2606.10546 Paper was published on June 09, 2026 This episode was AI-generated on June 11, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. When human experts write instruction documents for AI agents, pass rates jump sixteen points. When the model writes its own, the improvement is exactly zero — even though the documents look great. Microsoft's SkillAxe paper diagnoses why, with a fault-attribution trick that separates 'the instructions were bad' from 'the agent ignored good instructions' — and an honest look at how much the fix actually buys. Key Takeaways: - Why LLM-authored agent skills score zero improvement despite looking fluent and detailed — the valuable content is failure-derived trivia the model can't generate from general knowledge - How SkillAxe runs the agent twice (with and without the skill) and uses the skill's own stated rules as the grading rubric, so no external answer key is needed - The fault-attribution principle: an identical failure demands opposite repairs depending on whether the violated rule was precise enough to deserve following - The surprising decomposition: refined skills added nothing to per-attempt correctness — the entire gain came from helping the agent finish tasks at all, suggesting skills are institutional knowledge, not extra IQ - In the streaming SpreadsheetBench experiment, refinement bought compression and discoverability (22 skills loaded twice as often) rather than accuracy — the naive 69-skill library hit the same 52% pass rate - Where the headline claims weaken: under the benchmark's native scoring SkillAxe closes only ~11% of the gap to human skills, confidence intervals are wider than the effect, and the authors' own 'fair grader' is an LLM judge they built themselves 00:00 - The zero-improvement puzzle: Human-written skills lift agent pass rates by sixteen points, but LLM-authored skills — fluent and plausible-looking — help exactly as much as nothing. 03:47 - What a skill is, and why one bit of feedback isn't enough: Skills are runtime documentation the agent may consult or ignore, and a single pass/fail signal collapses four distinct failure modes — making naive refinement actively erode good content. 07:35 - The two-run differential diagnosis: SkillAxe runs each task with and without the skill and grades the difference against the skill's own rules, asking four questions: did it help, did it fire on the right tasks, was it followed, and does it cover all valid solution paths. 11:23 - Trigger geometry: measuring targeting with embeddings: Skill descriptions are plotted on a semantic map to check activation zones and exclusion boundaries — revealing that humans almost never write exclusion clauses, while refined skills end up with three-times-wider discrimination margins. 15:11 - Fault attribution: whose fault was the wrong shade of yellow?: The paper's centerpiece — when a rule is violated, the system asks whether the instruction was precise enough to deserve following, producing separate compliance and skill-quality scores instead of one muddled signal. 18:58 - Results: skills don't make agents smarter, they keep them from tripping: The headline 28% relative gain comes entirely from task completion, not correctness — illustrated by a Word placeholder trap that no amount of reasoning solves without procedural trivia. 22:46 - The flywheel experiment: compression, not accuracy: Streaming 200 tasks into a self-organizing skill library more than triples the bare agent's pass rate — but a naive 69-skill library matches it, so refinement's real win is fewer, sharper skills the agent actually loads. 26:34 - The steelman critique and the wrong-way flywheel: Tyler unpacks how the gap-closing claim shrinks from 67% to 11% under native scoring, the statistical power problem, the LLM-judge calibration issue, and the authors' own warning that imperfect diagnostics could bake errors into persistent documents. Recommended Reading: - Voyager: An Open-Ended Embodied Agent with Large Language Models: The skill-library agent Tyler cites directly in the episode as the contrast case — Voyager could refine its skills because Minecraft provided free verification, exactly the oracle SkillAxe has to do without. (https://arxiv.org/abs/2305.16291) - Self-Refine: Iterative Refinement with Self-Feedback: The foundational paper on having an LLM critique and rewrite its own outputs, which SkillAxe's evaluation-guided refinement loop extends from single responses to persistent skill documents. (https://arxiv.org/abs/2303.17651) - Large Language Models Cannot Self-Correct Reasoning Yet: A skeptical look at self-improvement without external feedback that directly echoes the episode's opening puzzle — why LLM-authored skills scored zero until a structured diagnostic signal was added. (https://arxiv.org/abs/2310.01798) - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: The key study of LLM-judge positional and calibration biases that underpins Tyler's central critique of SkillAxe's self-built 'fair grader' and judge-driven diagnostics. (https://arxiv.org/abs/2306.05685)