Let's verify step by step

The research explores two methods for improving large language models' ability to solve complex, multi-step mathematical problems: outcome supervision (OS), which provides feedback only on the final answer, and process supervision (PS), which offers feedback on each intermediate step. The authors demonstrate that process supervision significantly outperforms outcome supervision, particularly on challenging datasets like MATH, leading to more reliable models. They also introduce active learning as a method to enhance the efficiency of collecting human feedback for process supervision and release a large dataset, PRM800K, to support further research in this area. Ultimately, the paper argues that process supervision not only yields better performance but also promotes more interpretable and safer AI reasoning, highlighting its potential benefits for AI alignment.