EP149: [IDRBench] Interactive AI beats lone wolf models

The paper "IDRBench: Interactive Deep Research Benchmark" introduces the first systematic framework for evaluating interactive deep research conducted by Large Language Model (LLM) agents,. While existing systems typically operate autonomously, assuming a fully specified user intent, the authors argue that real-world research goals are often underspecified and evolve during the exploration process,.To address the limitations of existing benchmarks that only evaluate final outputs, IDRBench provides three core contributions:A Modular Multi-Agent Framework: This pipeline decomposes research into stages—Planning, Research Loop, and Generation—augmented with an explicit interaction mechanism for clarification and alignment,.Scalable User Simulation: A reference-grounded User Simulator acts as a proxy for human feedback, providing goal-oriented guidance based on reference documents to enable large-scale, reproducible evaluation without human annotators,.Interaction-Aware Evaluation: A comprehensive suite that jointly measures Interaction Benefits (such as semantic alignment and intent coverage) and Interaction Costs (measured in turns and tokens),,.Experiments conducted across seven state-of-the-art LLMs—including GPT-5.1, Gemini-2.5-Pro, and DeepSeek-V3.2—demonstrate that interaction consistently improves research quality and robustness,. Notably, the findings reveal that interaction can sometimes outweigh differences in raw model capacity, allowing lower-capacity models with effective interaction to surpass the autonomous performance of stronger models. The benchmark also highlights critical trade-offs between alignment gains and the operational overhead (cognitive and token costs) of frequent interaction,.