What Comes After AI Transformers? (Ep. 531)

The discussion sets the stage for exploring what comes after transformers.Key Points DiscussedTransformers show limits in reasoning, instruction following, and real-world grounding.The AI field is moving from scaling to exploring new architectures.Smarter transformers can be enhanced with test-time compute, neurosymbolic logic, and mixture-of-experts.Revolutionary alternatives like Mamba, Retinette, and world models introduce different approaches.Emerging ideas such as spiking neural networks, Kolmogorov Arnold networks, and temporal graph networks may reduce energy costs and improve reasoning.Neurosymbolic hybrids are highlighted as a promising path for logical reasoning.The challenge of commercializing research and balancing innovation with environmental costs.Hybrid futures likely combine multiple architectures into a layered system for AGI.The concept of swarm intelligence and agent collaboration as another route toward advanced AI.Timestamps & Topics00:00:00 💡 Introduction and GPT 5 disappointment00:02:00 🔍 The shift from scaling to new paradigms00:04:00 ⚙️ Smarter transformers and test-time compute00:05:20 🚀 Revolutionary alternatives including Mamba and Retinette00:06:20 🌍 World models and embodied AI00:06:58 🧠 Spiking neural networks and novel approaches00:11:00 ⛵ Exploration analogies and transformer context challenges00:12:20 🎮 Applications of world models in 3D spaces and XR00:16:45 🔗 Neurosymbolic hybrids for reasoning00:19:00 ⚡ Energy efficiency and productization challenges00:24:00 🌱 Balancing research speed with environmental costs00:31:00 📉 Four structural limits of transformers00:35:00 📚 RKV and new memory-efficient mechanisms00:37:00 📝 Analogies for architectures: note taker, stenographer, librarian, consultant00:41:00 🕵️ Transformer reasoning illusions and dangers00:44:00 🔬 Outlier experiments: physical neural nets, temporal graph networks, recurrent GANs00:49:00 🧩 Hybrid architecture visions for AGI00:53:30 🐝 Swarm agents and collaborative intelligence00:55:00 📢 Closing announcements and upcoming showsThe Daily AI Show Co-Hosts: Andy Halliday, Beth Lyons, Brian Maucere, Eran Malloch, Jyunmi Hatcher, and Karl Yeh