Efficient Exploration for LLMs

This Google DeepMind paper investigates efficient exploration strategies for improving large language models (LLMs) through reinforcement learning from human feedback (RLHF). The authors propose and evaluate various active exploration algorithms, contrasting them with passive methods. Their experiments, using a human preference simulator and the Gemini Nano model, demonstrate that active exploration, particularly using double Thompson sampling with epistemic neural networks (ENN) for uncertainty estimation, significantly reduces the number of human feedback queries needed to achieve high performance, potentially accelerating the path to superhuman ingenuity. They also highlight the crucial roles of both uncertainty estimation and the specific exploration scheme in achieving these benefits.