EP147: [DeepSynth-Eval] AI fails at deep research synthesis

The paper "DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing" introduces a new benchmark designed to address the lack of objective metrics for the post-retrieval synthesis stage of AI-driven research. While AI agents are increasingly used for "Deep Research," evaluating their ability to consolidate massive amounts of fragmented information into coherent, long-form reports has remained challenging due to the inherent subjectivity of open-ended writing.Key aspects of the paper include:DeepSynth-Eval (DSE) Benchmark: The authors created a benchmark consisting of 96 complex tasks derived from high-quality, expert-written survey papers. To isolate synthesis capability from retrieval performance, the benchmark provides an "Oracle Context" constructed from the original papers' bibliographies.Objective Checklist Metrics: The evaluation transforms subjective judgment into verifiable data by using two types of checklists: General Checklists for factual coverage and Constraint Checklists for structural organization (such as specific taxonomies or tables). This approach reduces "editorial freedom" to make model outputs more comparable to the gold-standard references.Experimental Findings: Results indicate that synthesizing information from hundreds of references is a "formidable open challenge," with even state-of-the-art (SOTA) models scoring below 40%.Workflow Insights: The study demonstrates that agentic "plan-then-write" workflows—which involve staged planning, reading, and iterative writing—significantly outperform single-turn generation. These multi-turn workflows effectively reduce hallucinations and improve a model's ability to follow complex structural instructions.Ultimately, the paper provides a reliable foundation for training and improving deep synthesis systems by offering a robust, reproducible standard for measuring long-form generation quality.