“Risk from fitness-seeking AIs: mechanisms and mitigations” by Alex Mallen

Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call "fitness-seeking"—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern. In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis. Fitness-seekers are, in many ways, notably safer than what I'll call "classic schemers". A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world's resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied[1] with modest-to-trivial costs. Still: Fitness-seekers risk some[2] of the same catastrophic outcomes as classic schemers do [...] ---Outline:(03:43) Overview(11:24) The basic reasons fitness-seekers might be safer than classic schemers(15:53) Four mechanisms for risk and their mitigations(16:38) Potemkin work(22:05) Instability(27:41) Manipulation(31:17) Outcome enforcement(36:44) Cross-cutting mitigations(37:10) Deals(39:57) Control(44:37) Alignment(44:40) Preventing fitness-seeking from arising(48:47) Making any fitness-seeking motivations safer(51:27) How does online training change the picture?(55:04) Overall recommendations(59:08) Conclusion The original text contained 18 footnotes which were omitted from this narration. --- First published: May 1st, 2026 Source: https://www.lesswrong.com/posts/9YCJZBtqr3FYL8rDp/risk-from-fitness-seeking-ais-mechanisms-and-mitigations --- Narrated by TYPE III AUDIO.