Provable and practical in-context policy optimization for self-improvement

This research paper introduces In-Context Policy Optimization (ICPO), a framework designed to explain and enhance the self-reflection capabilities of large language models. The authors provide a mathematical foundation proving that specific transformer architectures can inherently mimic policy optimization algorithms without requiring parameter updates. Building on this theory, they develop ME-ICPO, a practical algorithm that improves mathematical reasoning by iteratively refining responses based on self-assessed rewards. To ensure reliability, the system utilizes minimum-entropy selection and majority voting to filter out noise from self-evaluations. Empirical results demonstrate that this approach significantly boosts performance on complex reasoning benchmarks while remaining computationally efficient. Ultimately, the work bridges the gap between the theoretical understanding of in-context learning and the empirical success of test-time scaling.