Learning beyond Teacher

In this episode: • Distillation vs. Reinforcement Learning: Linda and Norris introduce the paper and discuss the differences between off-policy and on-policy distillation. • The Math Behind the Magic: The hosts dive into the mathematical equivalence between on-policy distillation and KL-constrained dense RL. • Extrapolating the Reward: Linda introduces Generalized OPD and the concept of reward extrapolation by setting the scaling factor above 1. • Merging Experts and Reward Correction: They discuss merging multiple domain teachers and the strong-to-weak distillation setting, including the cost of reward correction. • Conclusion and Future Outlook: Norris and Linda wrap up the episode with final thoughts on the practical applications and future of ExOPD.