Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

Researchers from Carnegie Mellon University introduced **Reasoning Cache (RC)**, a novel iterative decoding algorithm designed to help large language models solve complex, long-horizon problems. While standard reinforcement learning is often restricted by fixed training budgets, **RC** allows models to extrapolate their reasoning abilities to horizons over ten times longer than those seen during training. The method works by having the model generate a reasoning trace, **summarize** it into a "cache," and then discard the original trace to condition the next step on that summary. This approach exploits the **summarization-generation asymmetry**, where models are more effective at reasoning from a condensed history than from an exhaustive, repetitive log. In empirical tests, the **RCT-4B** model trained with this technique significantly outperformed much larger reasoning models on difficult mathematical and scientific benchmarks. Ultimately, **RC** provides a computationally efficient framework for scaling test-time compute, enabling models to refine and improve their solutions continually without being limited by training-time token constraints.