P(Doom) Estimates Shouldn't Inform Policy??
from Doom Debates! Hosted by Liron Shapira
Princeton Comp Sci Ph.D. candidate Sayash Kapoor co-authored a blog post last week with his professor Arvind Narayanan called "AI Existential Risk Probabilities Are Too Unreliable To Inform Policy". While some non-doomers embraced the arguments, I see it as contributing nothing to the discourse besides demonstrating a popular failure mode: a simple misunderstanding of the basics of Bayesian epistemology. I break down Sayash's recent episode of Machine Learning Street Talk point-by-point to analyze his claims from the perspective of the one true epistemology: Bayesian epistemology. 00:00 Introduction 03:40 Bayesian Reasoning 04:33 Inductive vs. Deductive Probability 05:49 Frequentism vs Bayesianism 16:14 Asteroid Impact and AI Risk Comparison 28:06 Quantification Bias 31:50 The Extinction Prediction Tournament 36:14 Pascal's Wager and AI Risk 40:50 Scaling Laws and AI Progress 45:12 Final Thoughts My source material is Sayash's episode of Machine Learning Street Talk: https://www.youtube.com/watch?v=BGvQmHd4QPE I also recommend reading Scott Alexander’s related post: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist Sayash's blogpost that he was being interviewed about is called "AI existential risk probabilities are too unreliable to inform policy": https://www.aisnakeoil.com/p/ai-existential-risk-probabilities Follow Sayash: https://x.com/sayashk Get full access to Doom Debates at lironshapira.substack.com/subscribe
Embed this episode
No transcript yet.