Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

Brendan O'Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped.Key topics- Discrete diffusion for text vs autoregressive generation- Why diffusion samples are more diverse, and what that unlocks for RL- Where diffusion already wins: latency, on-device, robotics- Why serving cost, not quality, is the real blocker- RL as the most underhyped area in AITimeline00:00 Introduction00:50 What diffusion models are and how text diffusion works04:40 Why Brendan bet on text diffusion in 202307:15 Diversity, creativity, and why it helps RL11:00 The best diffusion LLM today and the gap to frontier models14:25 Latency, serving cost, and why it needs more chips17:14 Where diffusion already wins: on-device, robotics, battery20:14 One model, two modes: diffusion for thinking, AR for answering22:24 Samplers and the stuttering problem26:27 Theory, BERT, and why now is a good time to work on this31:48 Pipelines built for autoregression, and continuous diffusion35:35 Hardware: flops vs bandwidth39:49 AGI timelines and real-world bottlenecks50:15 Is AI engineering or science?54:14 Most overhyped and most underhyped ideas58:35 RL on diffusion, value functions, and exploration1:07:30 Go download the model and break itMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.