Globally Convergent Offline Reinforcement Learning with Smoothed Bellman Residual Minimization

This paper introduces **Off-GLADIUS**, a novel algorithm designed for **offline reinforcement learning** that utilizes **Bellman Residual Minimization (BRM)**. While traditional BRM methods often struggle with stability and convergence issues, this research proves that the proposed approach achieves **global optimality** by satisfying a **Polyak–Łojasiewicz (PL) condition**. The authors establish that for linear and sufficiently wide **neural networks**, the algorithm converges linearly to the global optimum despite the non-convex nature of the objective function. This theoretical breakthrough addresses a long-standing open question regarding the convergence guarantees of gradient-based BRM in offline settings. Empirically, the study demonstrates that **Off-GLADIUS** matches or exceeds the performance of established baselines like **Conservative Q-Learning (CQL)** and **OptiDICE** across various control benchmarks. Ultimately, the paper bridges the gap between theoretical stability and practical effectiveness, offering a rigorous framework for learning optimal policies from fixed datasets.