Agent Learning via Early Experience

This paper discusses the "early experience" paradigm as a method for training autonomous language agents, aiming to bridge the gap between reward-free Imitation Learning (IL) and reward-dependent Reinforcement Learning (RL). This novel approach allows agents to learn from their own generated interactions, or "experience," without needing explicit external rewards, addressing a major challenge in real-world environments where dense feedback is often unavailable. The paper explores two core strategies within this paradigm: Implicit World Modeling (IWM), where the agent predicts future states to internalize environmental dynamics, and Self-Reflection (SR), where the agent compares its actions to expert demonstrations and generates rationales for superior choices. Experimental results across various benchmarks, including WebShop and ScienceWorld, consistently demonstrate that training with early experience significantly outperforms traditional imitation learning and provides a superior starting point, or "warm start," for subsequent reinforcement learning stages, even with reduced amounts of expert data.