“How risky would it be to make powerful AI obey one or a few people?” by cousin_it, Seth Herd

It seems fairly likely that the first powerful AIs will be instruction-following rather than value-aligned, and will be controlled by a small number of people. So it makes sense to worry what individual people might do with such immense power. Here intuitions diverge and careful analysis is scarce. This post presents a debate between Seth Herd and cousin_it over how risky such a scenario would be. The debate ran under an unusual protocol. First we wrote our initial draft statements and sent them to each other in private. Then we each revised our statements to strengthen them against the other's, and sent them to each other again. We continued this for about 10 rounds over the course of about a month, until we both agreed to stop revising and publish (while still remaining in disagreement). Here's the final pair of statements we ended up with, so you can judge for yourself: cousin_it's statement If there is an AI-assisted overlord (or several) and everyone else is their completely powerless subjects, that situation will be historically new, but not 100% new. Large power imbalances have existed in the past too and we can learn from them. Usually, when power was [...] ---Outline:(01:04) cousin_it's statement(05:55) Seth Herd's statement(07:30) Obedient ASI and human nature(09:29) Problems with distributed obedient AGI(11:18) Psychology and dynamics of secure unlimited power The original text contained 5 footnotes which were omitted from this narration. --- First published: August 11th, 2026 Source: https://www.lesswrong.com/posts/YtZBfbYRvMTynCfnC/how-risky-would-it-be-to-make-powerful-ai-obey-one-or-a-few --- Narrated by TYPE III AUDIO.