Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Maya and Leo dig into Anthropic's helpful-and-harmless RLHF paper: two opposite data-collection payrolls, one preference model serving two masters, the weekly online refresh that keeps the judge informed, the split-judge robustness test that exposes reward gaming, and a staged fight over whether the alignment tax is real. Sources: • Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback: https://arxiv.org/pdf/2204.05862 • Human preference data for Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback: https://github.com/anthropics/hh-rlhf • Training language models to follow instructions with human feedback: https://arxiv.org/abs/2203.02155