Do you really need to pretrain Q-functions for online RL fine-tuning?

Research from Stanford University challenges the conventional assumption that pre-training a Q-function on offline data improves reinforcement learning fine-tuning. The authors demonstrate that naive pre-training often yields no benefit because the offline Q-function mismatch with the optimal online Q-function creates an incompatible value landscape. To address this, they introduce Initialization via Policy Ensemble (IPE), a method that trains multiple diverse policies on the same data. By pooling rollouts from this policy ensemble, IPE provides broader action coverage and creates a more robust foundation for the critic. Experimental results across various robotic tasks show that IPE improves fine-tuning performance by an average of 26% over standard methods. This approach highlights that data diversity around the policy distribution is more critical for success than simply maximizing value during the offline phase.