µCODE: Code Generation with Single-Step Rewards

This document introduces µCODE, a novel approach for generating code iteratively based on execution feedback, departing from complex multi-turn reinforcement learning by leveraging the insight that code generation is a one-step recoverable process. µCODE employs a simple yet effective strategy, iteratively training a generator to propose code solutions and a verifier to evaluate them based on single-step rewards. This method not only demonstrates significant improvements over existing state-of-the-art techniques on code generation benchmarks but also offers a more stable and efficient training process. The paper provides theoretical analysis supporting the effectiveness of µCODE and explores the benefits of using a learned verifier for both training and optimizing inference through a multi-turn Best-of-N search strategy.