【第159期】TheAgentCompany：评估 AI 代理在真实工作场景中执行任务的新基准

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。 今天的主题是： TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks Summary TheAgentCompany is introduced as a new benchmark for evaluating AI agents on real-world workplace tasks. This benchmark simulates a software company environment where agents perform tasks like web browsing, coding, and communication with simulated colleagues. The paper assesses the performance of various large language models (LLMs) on these tasks, revealing that even the best models struggle to autonomously complete most of them. The authors identify challenges such as social interaction, navigating complex UIs, and the lack of training data for certain professional tasks. The benchmark aims to provide insights into the current capabilities and limitations of AI agents in automating work-related tasks. The benchmark also includes a breakdown of the employee roster of TheAgentCompany and examples of conversation between agents and simulated colleagues within their environment. The paper concludes by discussing the implications of their findings and suggesting directions for future research and benchmark improvements. TheAgentCompany 是一个用于评估 AI 代理在真实工作场景中执行任务的新基准测试。该基准模拟了一个软件公司环境，AI 代理需要完成 网页浏览、编写代码和与模拟同事沟通 等任务。论文评估了多种 大语言模型（LLMs） 在这些任务中的表现，结果表明，即使是最先进的模型仍难以自主完成大多数任务。研究指出了 社交交互、复杂 UI 导航 以及 某些专业任务缺乏训练数据 等关键挑战。 TheAgentCompany 旨在揭示 AI 代理在自动化工作任务中的当前能力与局限性。基准测试还包括公司员工角色的详细设定，以及 AI 代理与模拟同事之间的对话示例。论文最后讨论了研究结果的影响，并提出了未来研究方向及基准改进建议。 原文链接：https://arxiv.org/abs/2412.14161 前往小宇宙评论区与主播互动