The Illusion of the Illusion of Thinking

The provided text, a commentary on Shojaee et al. (2025), challenges claims that Large Reasoning Models (LRMs)exhibit fundamental reasoning failures on planning puzzles. The author argues that observed "accuracy collapse" stems from experimental design flaws rather than inherent model limitations. Key issues identified include models reaching output token limits and explicitly acknowledging this, evaluation frameworks misclassifying model capabilities by not distinguishing reasoning failures from practical constraints, and the inclusion of mathematically impossible puzzle instances that models are incorrectly penalized for not solving. The text suggests that alternative solution representations (like generating functions) restore performance, indicating that models possess the underlying algorithmic understanding but are constrained by output length requirements. Ultimately, the author emphasizes the importance of careful evaluation design to accurately assess AI reasoning.