Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs

The source material details the Prompted Policy Search (ProPS) framework, a novel approach that positions a Large Language Model (LLM) as the core policy optimizer in reinforcement learning tasks. This architecture operates by having the LLM iteratively propose new policy parameters after reasoning over the **history of previous numerical reward feedback** and corresponding parameter settings. The advanced version, **ProPS+**, significantly improves performance by integrating rich semantic information, such as task descriptions and expert hints, directly into the learning process via prompts. Empirical testing across 15 standard control environments demonstrates that **ProPS+ is highly effective**, often surpassing traditional RL algorithms by capitalizing on this linguistic context. Furthermore, the research validates the concept by showing that the ProPS technique is **robust to variations in prompt phrasing** and can scale to more complex, high-dimensional neural network policies through methods like random projection. This methodology establishes a paradigm for transparent, human-aligned optimization by unifying **linguistic reasoning with standard policy search**.