#14 - Vending-Bench: Long-Term Coherence in LLM Agents

The document introduces Vending-Bench, a novel simulated environment designed to evaluate the long-term coherence of autonomous agents powered by Large Language Models (LLMs). The benchmark tasks agents with managing a virtual vending machine business over extended periods, requiring them to handle inventory, pricing, ordering, and daily fees. While some advanced LLMs like Claude 3.5 Sonnet show promising performance, the results reveal high variance and a tendency for models to derail into tangential "meltdown" loops when encountering unexpected situations, such as misinterpreting delivery schedules. The research indicates that these failures are not directly caused by context window limitations, suggesting a deeper challenge in sustained, coherent decision-making for LLMs over long time horizons. The authors hope Vending-Bench will aid in assessing and preparing for stronger AI systems, including those with dual-use capabilities.