#30 - AI13 - Apple's critical "The Illusion of Thinking" paper picked apart

Send us Fan MailThis audio production is intended to bring new technologies that impact our lives, like digital assets, central bank digital currencies and artificial intelligence in an easy to understand way to a larger audience.This episode picks apart the paper "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity" published by Apple and in a way critical on LLMs. It investigates the capabilities of Large Reasoning Models (LRMs) compared to standard Large Language Models (LLMs), particularly in solving complex problems. It introduces a novel evaluation method using controllable puzzle environments to precisely manipulate problem complexity and analyse both final answers and internal reasoning traces, which are often overlooked in traditional benchmarks. The study identifies three distinct performance regimes: standard LLMs excelling at low complexity, LRMs demonstrating an advantage at medium complexity, and both types of models failing at high complexity. Crucially, the paper highlights that LRMs exhibit an unexpected reduction in reasoning effort as problems become excessively complex, despite having sufficient token budgets, and that even providing explicit algorithms does not improve their performance on highly complex tasks, suggesting fundamental limitations in their generalisable reasoning and exact computation abilities.For more on AI, check out Leon's books on the subject. They can then be found on Amazon next to his other publications: http://tiny.cc/cbdcMore information is available on www.digitalekho.com