EP141: [AIRS-Bench] AI agents beat human research benchmarks

This paper introduces AIRS-Bench (the AI Research Science Benchmark), a standardized suite of 20 tasks designed to rigorously evaluate the capabilities of AI agents as autonomous research scientists. Developed by researchers at FAIR at Meta in collaboration with the University of Oxford and University College London, the benchmark is curated from state-of-the-art (SOTA) machine learning papers to ensure the tasks are both challenging and relevant.Key aspects of the research include:Comprehensive Evaluation: AIRS-Bench assesses agents across the full research lifecycle, including idea generation, methodology design, implementation, experiment analysis, and iterative refinement.Challenging Methodology: Agents are required to generate the code necessary to train and validate machine learning models without access to baseline code, reflecting a realistic research workflow.Diverse Domains: The benchmark covers seven distinct categories: language modeling, mathematics, code generation, molecular and protein modeling, and time-series forecasting.Empirical Findings: The researchers evaluated 14 agent configurations using frontier models (such as GPT-4o and o3-mini) paired with different "scaffolds" (linear and parallel search algorithms). The results showed that while agents surpassed human SOTA in four tasks, they failed to match it in sixteen others.Unsaturated Results: Even in cases where agents exceeded human benchmarks, they did not reach the theoretical performance ceilings, indicating that the benchmark is far from solved and has significant headroom for future development.The authors have open-sourced the task definitions and evaluation code to catalyze the development of more advanced agents capable of accelerating scientific progress.