Reasoning with Sampling: Base Models Outperform RL

The research paper titled "Reasoning with Sampling: Your Base Model is Smarter Than You Think" by Harvard researchers introduces a novel, training-free iterative sampling algorithm inspired by Markov Chain Monte Carlo (MCMC) techniques to enhance the reasoning capabilities of large language models (LLMs) at inference time. This method, termed "Power Sampling," leverages the base model's own likelihoods to simulate sampling from a "power distribution," which sharpens the distribution toward higher-likelihood sequences without additional training or the need for a reward signal. The authors argue that this technique successfully elicits latent reasoning skills in base models, demonstrating performance on par with, and sometimes exceeding, models post-trained with Reinforcement Learning (RL), particularly the Group Relative Policy Optimization (GRPO) method, across diverse benchmarks like MATH500, HumanEval, and GPQA. Crucially, Power Sampling maintains greater generation diversity compared to RL-posttraining, which typically suffers from a collapse in multi-shot performance.