State-Adaptive Regularization for Offline Reinforcement Learning

This research introduces a novel selective state-adaptive regularization method for offline reinforcement learning (RL), which aims to learn effective policies from static datasets. Unlike previous approaches that use uniform regularization, this method dynamically adjusts regularization strength across different states, recognizing variations in data quality. By establishing a connection between value regularization (like CQL) and explicit policy constraint methods, the approach extends its applicability to both. Furthermore, it incorporates a selective regularization strategy that prioritizes high-quality actions to enhance performance, particularly in datasets with varying data quality. Experimental results demonstrate that this method significantly outperforms existing state-of-the-art techniques in both offline and offline-to-online settings, fostering more efficient fine-tuning.