Scaling Test-Time Compute for Reasoning Models

This episode explores how test-time compute should be allocated in large language models, using a recent study that compares parallel sampling, majority voting, shortest- and longest-trace selection, and beam-style search under a common evaluation setup. It explains the paper’s central argument that there is no single best inference-time strategy: some model families behave like short-horizon reasoners that benefit from several concise attempts, while others act like long-horizon reasoners that can make productive use of longer sequential reasoning. The discussion also examines how the authors benchmark eight open models across demanding datasets such as AIME and GPQA Diamond, and why harder problems reveal whether extra trace length produces real progress or just more verbose failure. Listeners would find it interesting because it turns a vague idea of “letting models think longer” into a concrete engineering question about how reasoning systems should spend their runtime budget. Sources: 1. The Art of Scaling Test-Time Compute for Large Language Models — Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty, 2025 http://arxiv.org/abs/2512.02008 2. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024 https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters 3. Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning — Michael Hassid, Gabriel Synnaeve, Yossi Adi, Roy Schwartz, 2025 https://scholar.google.com/scholar?q=Don%27t+Overthink+it.+Preferring+Shorter+Thinking+Chains+for+Improved+LLM+Reasoning 4. Inverse Scaling in Test-Time Compute — Aryo Pradipta Gema, Alexander Hagele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, Ethan Perez, 2025 https://scholar.google.com/scholar?q=Inverse+Scaling+in+Test-Time+Compute 5. Qwen3 Technical Report — An Yang and the Qwen team, 2025 https://scholar.google.com/scholar?q=Qwen3+Technical+Report 6. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang et al., 2023 https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models 7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao et al., 2023 https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models 8. Graph of Thoughts: Solving Elaborate Problems with Large Language Models — Maciej Besta et al., 2024 https://scholar.google.com/scholar?q=Graph+of+Thoughts%3A+Solving+Elaborate+Problems+with+Large+Language+Models 9. short-m@k — Ranit Hassid et al., 2025 https://scholar.google.com/scholar?q=short-m%40k 10. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — approx. recent process-verifier work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=Rewarding+Progress%3A+Scaling+Automated+Process+Verifiers+for+LLM+Reasoning 11. Improving LLM Reasoning Through Scaling Inference Computation With Collaborative Verification — approx. recent collaborative-verification work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=Improving+LLM+Reasoning+Through+Scaling+Inference+Computation+With+Collaborative+Verification 12. Graph of Verification: Structured Verification of LLM Reasoning With Directed Acyclic Graphs — approx. recent verification-structure work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=Graph+of+Verification%3A+Structured+Verification+of+LLM+Reasoning+With+Directed+Acyclic+Graphs 13. Dynamic Parallel Tree Search for Efficient LLM Reasoning — approx. recent tree-search work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=Dynamic+Parallel+Tree+Search+for+Efficient+LLM+Reasoning 14. Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfalls — approx. recent tree-search analysis work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=Don%27t+Get+Lost+in+the+Trees%3A+Streamlining+LLM+Reasoning+by+Overcoming+Tree+Search+Exploration+Pitfalls 15. REST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search — approx. recent MCTS/process-reward work, exact authors not confirmed from snippet, 2025 https://scholar.google.com/scholar?q=REST-MCTS%2A%3A+LLM+Self-Training+via+Process+Reward+Guided+Tree+Search 16. Large Language Models Cannot Self-Correct Reasoning Yet — approx. recent self-correction evaluation work, exact authors not confirmed from snippet, 2024 https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet 17. AI Post Transformers: Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025 https://podcast.do-not-panic.com/episodes/test-time-scaling/ 18. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp3 19. AI Post Transformers: Agentic Aggregation for Long-Horizon AI Tasks — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-22-agentic-aggregation-for-long-horizon-ai-4c1a71.mp3 20. AI Post Transformers: TUMIX Multi-Agent Test-Time Scaling with Tools — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-22-tumix-multi-agent-test-time-scaling-with-40671c.mp3 21. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3 22. AI Post Transformers: The Art of Scaling Reinforcement Learning Compute for LLMs — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/the-art-of-scaling-reinforcement-learning-compute-for-llms/