ACPBench Hard: Generative Planning Reasoning Tasks

The provided paper introduces ACPBench Hard, a new benchmark designed to evaluate the reasoning capabilities of large language models for automated planning. This benchmark extends the original ACPBench by featuring open-ended, generative versions of planning-related questions across various tasks, mirroring the challenges faced by symbolic planners. The authors tested several large language and reasoning models, including state-of-the-art options, on ACPBench Hard and found that their performance was generally subpar, indicating a significant gap in their ability to reliably reason about planning. The research highlights the difficulty current models face with core planning reasoning tasks and suggests future directions for improving their performance.