“Catastrophe through Chaos” by Marius Hobbhahn

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. Many other people have talked about similar ideas, and I claim neither novelty nor credit. Note that this reflects my median scenario for catastrophe, not my median scenario overall. I think there are plausible alternative scenarios where AI development goes very well.When thinking about how AI could go wrong, the kind of story I’ve increasingly converged on is what I call “catastrophe through chaos.” Previously, my default scenario for how I expect AI to go wrong was something like Paul Christiano's “What failure looks like,” with the modification that scheming would be a more salient part of the story much earlier. In contrast, “catastrophe through chaos” is much more messy, and it's much harder to point to a single clear thing that went wrong. The broad strokes of [...] ---Outline:(02:46) Parts of the story(02:50) AI progress(11:12) Government(14:21) Military and Intelligence(16:13) International players(17:36) Society(18:22) The powder keg(21:48) Closing thoughts--- First published: January 31st, 2025 Source: https://www.lesswrong.com/posts/fbfujF7foACS5aJSL/catastrophe-through-chaos --- Narrated by TYPE III AUDIO.