SPIRAL: Learning to search and aggregate

The Spiral framework addresses a limitation in current language model training where models are optimized for single-trace reasoning but fail to coordinate complex inference strategies at test time. To solve this, researchers combine set reinforcement learning with standard reinforcement learning to train models on sequential, parallel, and aggregative compute primitives simultaneously. The model learns to generate a diverse set of parallel search traces that are specifically designed to be synthesized by a downstream aggregator into a correct final response. By optimizing the entire pipeline end-to-end, the system moves beyond rigid, hand-designed scaffolds toward learned search procedures. Experimental results demonstrate that this method significantly improves scaling efficiency and performance on difficult mathematical reasoning tasks. Ultimately, Spiral enables models to effectively utilize larger token budgets through recursive self-aggregation and more sophisticated verification behaviors.