“Motivated reasoning, confirmation bias, and AI risk theory” by Seth Herd

Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias. - From Scott Alexander's review of Julia Galef's The Scout Mindset. Alexander goes on to argue that this bias is the source of polarization in society, which is distorting our beliefs and setting us at each other's throats. How could someone believe such different things unless they're either really stupid or lying to conceal their selfishness? I think smart people who care about the truth go on believing conflicting things largely because of confirmation bias and motivated reasoning. The corner of civilization I'm most worried about is the one figuring out how to handle the advent of strong AI. I'm not telling anyone which direction to update, but I am suggesting that we are probably a little to a lot overconfident in our beliefs about alignment and AI impacts. I think the effects of biases are still strong and still overlooked in this corner of civilization, despite its strong values of truth-seeking and relative awareness of biases. Bias has more influence where there's less direct evidence, and that's the case in [...] ---Outline:(04:50) 1.1. Motivated reasoning(10:01) 2. Empirical evidence for confirmation bias(12:08) 2.1. Bias in evaluating evidence(17:48) 2.2. Bias in selecting evidence(20:57) 2.3. Bias in remembering evidence(22:37) 2.4. Other causal explanations of confirmation bias effects(25:17) 2.5. Empirical evidence for motivated reasoning(27:26) 3. Limitations in human cognitive capacity for very complex problems(29:16) 3.1. Introspection suggests fuzzy models and updating(30:32) 3.2. Intuition vs. analysis - evidence and brain mechanisms(34:19) 3.3. Bayesian reasoning is an ideal, not a method(36:40) 3.4. AI risk is complicated(40:18) 4. Compounding of confirmation bias(42:05) 4.1. Example of frame/hypothesis choices and confident disagreement among experts(47:29) 4.2. Social compounding of confirmation bias effects(49:45) 4.2.1. Social effects on evaluating evidence.(52:41) 4.2.2. Social effects on selecting evidence, memory, and framing(57:48) 4.2.3. Interlude: dont give up on seeking truth(58:32) 4.2.4. Social belief contagion or information cascade effects(01:03:07) 4.3. Very rough estimates of total compounded confirmation bias(01:09:01) 5. Implications and remediations(01:12:02) 5.1. Standard remediations(01:14:09) 5.2. Remediations for motivated reasoning(01:17:39) Conclusion The original text contained 9 footnotes which were omitted from this narration. --- First published: May 5th, 2026 Source: https://www.lesswrong.com/posts/QpgmEhBvJQxAfFMP2/motivated-reasoning-confirmation-bias-and-ai-risk-theory --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.