What Actually Matters in AI? - with Zhuang Liu (Princeton)

In this episode, we hosted Zhuang Liu, Assistant Professor at Princeton and former researcher at Meta, for a conversation about what actually matters in modern AI and what turns out to be a historical accident.Zhuang is behind some of the most important papers in recent years (with more than 100k citations): ConvNeXt (showing ConvNets can match Transformers if you get the details right), Transformers Without Normalization (replacing LayerNorm with dynamic tanh), ImageBind, Eyes Wide Shut on CLIP's blind spots, the dataset bias work showing that even our biggest "diverse" datasets are still distinguishable from each other, and more.We got into whether architecture research is even worth doing anymore, what "good data" actually means, why vision is the natural bridge across modalities but language drove the adoption wave, whether we need per-lab RL environments or better continual learning, whether LLMs have world models (and for which tasks you'd need one), why LLM outputs carry fingerprints that survive paraphrasing, and where coding agents like Claude Code fit into research workflows today and where they still fall short.Timeline00:13 — Intro01:15 — ConvNeXt and whether architecture still matters06:35 — What actually drove the jump from GPT-1 to  GPT-308:24 — Setting the bar for architecture papers today11:14 — Dataset bias: why "diverse" datasets still aren't22:52 — What good data actually looks like26:49 — ImageBind and vision as the bridge across modalities29:09 — Why language drove the adoption wave, not vision32:24 — Eyes Wide Shut: CLIP's blind spots34:57 — RL environments, continual learning, and memory as the real bottleneck43:06 — Are inductive biases just historical accidents?44:30 — Do LLMs have world models?48:15 — Which tasks actually need a vision world model50:14 — Idiosyncrasy in LLMs: pre-training vs post-training fingerprints53:39 — The future of pre-training, mid-training, and post-training57:57 — Claude Code, Codex, and coding agents in research59:11 — Do we still need students in the age of autonomous research?1:04:19 — Transformers Without Normalization and the four pillars that survived1:06:53 — MetaMorph: Does generation help understanding, or the other way around?1:09:17 — WrapMusic:"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0."Palms Down" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.Changes: trimmedAbout: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.