Entropy and Reinforcement Learning for LLMs

This academic paper explores a critical issue in reinforcement learning (RL) with large language models (LLMs): the rapid decline of policy entropy, which limits the models' ability to explore and improve. The authors demonstrate an empirical relationship where performance gains are directly tied to entropy reduction, leading to a predictable performance ceiling. To address this, they analyze the dynamics of policy entropy, showing its change is linked to the covariance between action probability and advantage. Based on this understanding, the paper proposes two novel techniques, Clip-Cov and KL-Cov, which effectively manage entropy by restricting updates to high-covariance tokens, thus promoting continuous exploration and achieving superior performance in reasoning tasks.