MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

MLE-bench is a benchmark that evaluates the performance of AI agents on machine learning engineering tasks. The benchmark is comprised of 75 real-world Kaggle competitions, each with a dataset, description, and grading code. The authors evaluated various language models and agent frameworks on MLE-bench, finding that the best-performing agent achieved at least the level of a Kaggle bronze medal in 16.9% of the competitions. The paper discusses various ways to improve agent performance, such as increasing the number of attempts and the amount of compute available. It also explores potential contamination issues that might affect the benchmark's results. The benchmark is open-source and aims to promote research in understanding the capabilities of agents for automating ML engineering.