Surya Ganguli: The Physics of Intelligence

Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.key topicsConnections between physics, neuroscience, and AIEmergent properties in complex systemsScaling laws in language modelsData efficiency and pruning in AINeuroscience insights into consciousness and selfThe future of AI and brain modelingChapters00:00 Introduction to Surya Ganguli 00:57 Surya's Background: From String Theory to Neuroscience02:22 Emergent Properties in Physics, Neuroscience, and AI03:16 Energy Landscapes and Loss Landscapes in High Dimensions04:07 Why Local Minima Don't Exist in High-Dimensional AI05:22 Gradient-Based vs. Gradient-Free Learning Methods08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges13:48 Scaling Laws and Data Efficiency in Language Models18:10 Properties of Data that Affect Scaling Laws22:04 Constructing Non-Redundant Data Sets for Better Learning24:32 Theory vs. Empirical Results in AI Research32:19 Fundamental Components of Deep Learning: Are They Changing?34:31 Future Paradigms in AI Beyond Current Models37:22 Teaching AI and Humans: Paradigm Shifts in Learning41:37 Consciousness, Self, and the Brain: Surya's Perspectives49:49 Neuroscience and AI: Understanding the Brain and Consciousness01:02:03 Understanding the Brain: Challenges and Opportunities01:09:21 Brain-Computer Interfaces and AI in NeuroscienceMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.