EPISODE · Jun 20, 2026 · 29 MIN
How LLMs Actually Learn: Stages or Slurry?
from My Weird Prompts
When you train a large language model from scratch, does it learn in stages — grammar first, then facts, then reasoning? Or does everything bloom in parallel? The honest answer is both. This episode unpacks what loss curves actually reveal about training dynamics: how syntax and factual knowledge race each other from day one, why "grokking" complicates the picture, and how the training data's natural distribution creates a self-curriculating curriculum. We explore why the model's internal representations crystallize over time, how emergent abilities appear at scale thresholds, and why the training process is best understood as a single undifferentiated slurry of next-token prediction from which structure emerges.
What this episode covers
When you train a large language model from scratch, does it learn in stages — grammar first, then facts, then reasoning? Or does everything bloom in parallel? The honest answer is both. This episode unpacks what loss curves actually reveal about training dynamics: how syntax and factual knowledge race each other from day one, why "grokking" complicates the picture, and how the training data's natural distribution creates a self-curriculating curriculum. We explore why the model's internal representations crystallize over time, how emergent abilities appear at scale thresholds, and why the training process is best understood as a single undifferentiated slurry of next-token prediction from which structure emerges.
NOW PLAYING
How LLMs Actually Learn: Stages or Slurry?
No transcript for this episode yet
Similar Episodes
Mar 26, 2026 ·1m
Jan 2, 2026 ·47m
Dec 21, 2025 ·46m