Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1

Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1 Source: https://arxiv.org/abs/2604.06506 Paper was published on April 07, 2026 This episode was AI-generated on May 3, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A frontier coding agent given full access to ten major open-source projects found twelve security bugs. A constrained pipeline using the same model class found three hundred seventy-nine. The gap isn't about compute — it's an argument about where LLMs actually belong in a rigorous engineering stack. Key Takeaways: - Why symbolic execution has been 'almost practical' for fifty years, and what specifically was blocking it from going mainstream - The architectural move at the heart of SAILOR: the LLM writes the test harness, but never gets to declare a bug — deterministic tools do - Why iteration matters so much: removing the feedback loop drops confirmed bugs from 379 to zero - The three projects where SAILOR found nothing (curl, OpenSSL, SQLite) and what that tells you about which codebases this approach fits - Why 40% of the bugs found are essentially invisible to standard fuzzing, and what that means for the current state of automated security testing - A general pattern for deploying LLMs in serious engineering work: route every model output through tools whose failure modes are independent of the model's 00:00 - The 12-versus-379 result: Setting up the headline comparison between a full coding agent and SAILOR's constrained pipeline on the same target codebases. 04:01 - Why symbolic execution never went mainstream: The harness-writing bottleneck that has kept a mathematically beautiful technique sidelined for half a century. 08:03 - Epistemic decomposition: detective, locksmith, forensics lab: The three-component architecture that assigns each tool exactly one question — where, how, and whether — and forbids it from answering the others. 12:05 - A real bug from start to finish: Following one heap buffer overflow in GNU Binutils through CodeQL flagging, LLM harness-writing with iterative feedback, and AddressSanitizer confirmation. 16:07 - What the bug counts actually look like: Per-project breakdowns across mupdf, FFmpeg, libpng, and others — and why 40% of the findings are essentially unreachable by fuzzing. 20:09 - The honest limitations: Steelmanning the result: the 0.5% confirmation rate, three projects that returned zero bugs, the gap between memory-safety bug and exploitable vulnerability, and deduplication caveats. 24:11 - Why the pattern generalizes: The broader architectural argument — let LLMs generate scaffolding, but route every claim through deterministic tools whose failure modes don't share the model's. 28:13 - What's next and what to watch: Where the same template might apply to fuzzing harnesses and formal verification, and the kinds of bugs that won't decompose into this structure. Recommended Reading: - KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs: The foundational symbolic execution engine paper that defines the 'precision instrument' SAILOR's LLM-written harnesses are designed to drive. (https://llvm.org/pubs/2008-12-OSDI-KLEE.html) - Fuzzing: Hayes, Miller, et al. — A Survey of Symbolic Execution Techniques: A comprehensive survey of why symbolic execution has been 'almost practical for fifty years,' giving context for the harness-writing bottleneck SAILOR targets. (https://arxiv.org/abs/1610.00502)