Activation Steering in Generative Settings via Contrastive Causal Mediation Analysis

This academic paper introduces Contrastive Causal Mediation (CCM), a novel and computationally efficient method for identifying and intervening on the internal activations of large language models (LLMs) to control their free-form text generation. Traditional causal mediation analysis struggles with free-form text outputs, so CCM proposes using the difference in generation probabilities between contrastive response pairs (successful vs. unsuccessful steering) as a robust signal for localization. The researchers apply CCM to three challenging behavioral control tasks—refusal, sycophancy, and style transfer—across several LLMs, demonstrating that their method consistently outperforms existing probing and random baselines in pinpointing the most effective attention heads for steering. The study concludes that this causally grounded approach to mechanistic interpretability shows great promise for fine-grained model control at inference time.