Beyond Reward Hacking: Causal Rewards for Large LanguageModel Alignment

This research introduces a novel method for aligning large language models (LLMs) with human preferences while avoiding common pitfalls like reward hacking and spurious correlations. The authors propose a causal reward modeling approach that integrates causal inference and counterfactual invariance to ensure that reward predictions are based on true relationships rather than irrelevant data patterns. Through experiments on various datasets, including those focused on sycophancy, length, concept, and discrimination biases, they demonstrate that this method effectively mitigates these issues. The paper highlights that this causal reward modeling is a practical enhancement that can be seamlessly integrated into existing RLHF workflows to improve the trustworthiness and fairness of LLM finetuning.