EP046: Training AI With A Constitution

The paper "Constitutional AI: Harmlessness from AI Feedback" by Anthropic introduces a method to train AI systems to be helpful and harmless without relying on human feedback labels to identify harmful outputs. The core concept, termed Constitutional AI (CAI), governs the AI's behavior using a short list of natural language rules or principles—referred to as a "constitution".The CAI training process involves two main stages:Supervised Learning (SL) via Self-Critique and Revision: The system prompts a helpful-only AI to generate responses to harmful queries. The AI is then asked to critique its own toxic response based on a randomly selected principle from the constitution and revise the response to remove harmful content. A pretrained model is then finetuned on these self-revised, harmless responses.Reinforcement Learning from AI Feedback (RLAIF): The SL-trained model generates pairs of responses to harmful prompts. Instead of using human evaluators, an AI acts as the judge, evaluating which response is better according to the constitutional principles. A preference model is trained on this AI-generated feedback, and the final model is fine-tuned against it using reinforcement learning.Key Results and Outcomes:Non-Evasive Harmlessness: Unlike previous models that simply refused to answer controversial questions or shut down conversations, the CAI model is designed to be non-evasive. It engages with harmful queries by thoughtfully explaining its ethical objections rather than dodging the prompt.Scaling AI Supervision: The paper demonstrates that as language models become more capable, they can effectively replace thousands of human preference labels by supervising other AIs.Transparency: The process leverages Chain-of-Thought (CoT) reasoning, which improves the AI's ability to identify harms and makes its decision-making process more explicit and transparent during training.