Reinforcement Learning from Human Feedback: Progress and Challenges

This episode closes Topic 5 with John Schulman's Berkeley EECS colloquium on RLHF progress and challenges — the architect of PPO and ChatGPT-era preference tuning grading his own pipeline. Maya and Leo walk the progress column (comparisons make negative feedback usable where intuition outruns specification) and four challenge landmarks: the Applause Meter, the Tired Jury, the Smooth Talker, and the First Monday. They stage the field's central argument — breakthrough interface versus rater-satisfaction proxy — and settle it as sensor-versus-actuator claims that only independent behavioral audits can adjudicate. Sources: • Reinforcement Learning from Human Feedback: Progress and Challenges: https://eecs.berkeley.edu/research/colloquium/230419-2/ • Training Language Models to Follow Instructions with Human Feedback: https://arxiv.org/abs/2203.02155 • RLHF: Reinforcement Learning from Human Feedback: https://huyenchip.com/2023/05/02/rlhf.html • Deep Reinforcement Learning from Human Preferences: https://arxiv.org/abs/1706.03741