SPIRAL: Self-Play for Reasoning in Games

The research introduces SPIRAL, a novel self-play framework for Large Language Models (LLMs) that fosters advanced reasoning abilities without relying on human-curated data or complex reward engineering. By engaging LLMs in multi-turn, zero-sum games against continuously improving versions of themselves, SPIRAL generates an infinite curriculum of challenging problems. The paper highlights that this self-play approach, enhanced by Role-conditioned Advantage Estimation (RAE) to stabilize training, leads to transferable reasoning skills that significantly boost performance on unrelated mathematical and general reasoning benchmarks. The study demonstrates how different games cultivate specific cognitive patterns, and how multi-game training synergistically combines these strengths, proving that competitive game environments can serve as effective "reasoning gymnasiums" for LLMs.