Spectral Bellman Method: Unifying RL Representation and Exploration

This paper introduces the Spectral Bellman Method (SBM), a novel framework designed to enhance value-based reinforcement learning by unifying representation learning and exploration. By leveraging the Inherent Bellman Error (IBE) condition, the authors demonstrate that optimal feature representations are intrinsically linked to the spectral properties of the Bellman operator. This theoretical connection allows the agent to learn state-action features whose covariance structure is naturally aligned with environment dynamics, facilitating more effective Thompson Sampling for exploration. Empirical evaluations on the Atari benchmark show that SBM significantly improves performance in hard-exploration and long-horizon tasks when integrated into standard algorithms like DQN and R2D2. Ultimately, the method offers a computationally tractable and principled approach to achieving Bellman consistency across a broad space of value functions.