“Deconfusing Corrigibility” by berns

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations. Introduction Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.) Here, my intention is to clear up some of these confusions and present a new picture of corrigibility that builds on these existing accounts to specify a goal that is more congruous with the original 2015 desiderata, and admits [...] --- Outline: (00:34) Introduction (02:38) From normative learning to corrigibility (02:43) The complete normative learning problem (06:55) Three kinds of uncertainty (09:48) Dangers of a complete normative goal (13:19) An analogy from human relationships (17:12) The two component goals of corrigibility (18:50) Confusions in existing pictures of corrigibility (18:55) Over-epistemicized corrigibility (20:28) Fully updated deference (23:50) Inputs to the utility function (26:26) Emergent corrigibility is not enough (28:43) Why not a singular target (31:01) Evaluatively-thickened corrigibility (33:45) Unnatural reasoning in singular-target agents (40:03) A new picture of corrigibility (40:07) Allocating authority (44:36) Effective exercise of reserved powers (47:27) Latency and the cost of non-delegation (49:38) Operationalizing Goal 2 (56:10) Legitimacy (01:00:37) A formalism for corrigibility (01:01:01) Definitions (01:04:35) Prioritization under certainty (01:06:18) Prioritization under uncertainty (01:14:11) Learning with logical induction (01:25:43) Limitations and extensions (01:25:47) The state of the corrigibility model (01:26:42) Who is the human principal? (01:32:14) Instilling goals in practical training (01:33:50) A thicker concept of legitimacy The original text contained 17 footnotes which were omitted from this narration. --- First published: October 17th, 2026 Source: https://www.lesswrong.com/posts/f8R3B2zqqTJZjSAsm/deconfusing-corrigibility --- Narrated by TYPE III AUDIO.