Skip to content
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective episode artwork

May 16, 2025 17 min

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective

from Best AI papers explained Hosted by Enoch H. Kang

This research explores ways to make Reinforcement Learning from Human Feedback (RLHF) more sample-efficient by leveraging imperfect reward models. The authors identify a key property of the KL-regularized RLHF objective, showing that a policy's ability to cover the optimal policy is linked to its sub-optimality, which suggests that higher policy value indicates better coverage. Building on this insight, they propose a novel transfer learning approach and a theoretically-sound algorithm, Transfer Policy Optimization (TPO), which uses a policy-value-based transfer selection strategy and incorporates "self-transfer learning" from data collected during the online process. They also develop a more practical empirical TPO algorithm that uses win rates for policy selection to reduce computational costs and demonstrate its effectiveness on summarization tasks.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective

0:00 17:54

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Best AI papers explained?

This episode is 17 minutes long.

When was this Best AI papers explained episode published?

This episode was published on May 16, 2025.

Can I download this Best AI papers explained episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.