How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)

Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability.LINKS:Nora Ammann siteARIA safeguarded AI program pageAI Resilience official siteGradual Disempowerment websiteCHAPTERS:(00:00) Episode Preview(01:00) Slow takeoff expectations(08:13) Domination versus chaos(17:18) Human-AI coalitions vision(28:14) Scaling oversight and agents(38:45) Formal specs and guarantees(51:10) Resilience in AI era(01:02:21) Defense-favored cyber systems(01:10:37) AI-enabled bargaining and tradePRODUCED BY:https://aipodcast.ingSOCIAL LINKS:Website: https://podcast.futureoflife.orgTwitter (FLI): https://x.com/FLI_orgTwitter (Gus): https://x.com/gusdockerLinkedIn: https://www.linkedin.com/company/future-of-life-institute/YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/Apple: https://geo.itunes.apple.com/us/podcast/id1170991978Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP