Scaling Self-Play with Self-Guidance

This paper discusses Self-Guided Self-Play (SGS), a new algorithm designed to improve the reasoning capabilities of large language models through autonomous problem generation. Standard self-play often hits a performance plateau because the Conjecturer model eventually creates low-quality or "hacked" problems that do not facilitate real learning for the Solver. To solve this, SGS adds a Guide role that evaluates synthetic tasks for elegance and relevance to target goals, ensuring the training data remains high-quality over hundreds of rounds. This three-part system of Solver, Conjecturer, and Guide allows models to sustain improvement for significantly longer periods than previous methods. Testing on formal mathematical theorem proving in Lean4 shows that a 7B parameter model using SGS can eventually outperform much larger models. The research emphasizes that managing model entropy and providing structured guidance are essential for scaling reinforcement learning effectively.