“Extreme concentration of power over ASI has non-obvious advantages” by Seth Herd

This post is an extension of a collaboration with cousin_it on the question How risky would it be if powerful AI obeyed one or a few people?. There he argues for a common position: a future controlled by one or a few humans with powerful AI aligned to their intent is likely to produce terrible outcomes. My position is guardedly optimistic, for reasons I think are fairly novel: humans tend strongly to be better and become better over time under good circumstances, and near-perfect power and knowledge are the best circumstances. That post contains his essay and the abstract and overview sections of this post as my shorter response. This piece grew longer than our original target, because the subject is potentially critical for alignment strategy, and has not been analyzed in any depth, to my knowledge. Abstract: Concentration of power over AGI/ASI seems quite possible. The first AGIs being aligned to intent (or instructions) over values seems fairly likely. So one or a few individuals or small groups gaining power over ASI seems fairly likely. Thus it seems relevant to technical alignment strategy (value alignment vs. corrigibility) to worry about what individuals might do with such [...] ---Outline:(02:20) 1. Overview(02:24) 1.1. Obedient ASI and human nature(04:22) 1.2. Problems with distributed obedient AGI(06:14) 1.3. Psychology and dynamics of secure unlimited power(09:24) 2. Historical evidence does not directly apply, since power hasn't yet been secure or absolute(10:31) Late Russian serfdom as a historical example(13:16) 2.1. Incompetence, ignorance, and greed are the causes of most suffering under dictatorships(15:31) 3. Outcomes(17:33) Additional considerations on outcome predictions(22:24) 4. Risks of human-controlled singleton ASI(24:13) 5. Does widely distributed human-controlled AGI reduce or increase risk?(25:44) 5.1. Problems with defending against many AIs each capable of creating new offenses(29:58) 5.2. The case for optimism about distributed power over AGI/ASI(31:01) The analogy to modern power distribution(33:17) New AI-enabled paths to stable power distribution(34:49) 6. Conclusion The original text contained 11 footnotes which were omitted from this narration. --- First published: August 11th, 2026 Source: https://www.lesswrong.com/posts/h3eHNerYRmtvoi8cF/extreme-concentration-of-power-over-asi-has-non-obvious --- Narrated by TYPE III AUDIO.