EP 33 | CME295: LLM Training
from AI Bites: The Academic Series Hosted by Jack Lakkapragada
Building a massive language model from scratch requires an astronomical amount of data and compute. In this episode, we explore the "pre-training" phase. We break down the sheer scale of the engineering required to teach a model the fundamental statistical rules of human language. Key Topics: The Data Diet: How trillions of words are scraped, cleaned, and curated to form the foundation of an LLM's knowledge. Tokenization: How models break down human words into numbers they can actually process and predict. Next-Token Prediction: The core objective function. How simply guessing the next word in a sequence builds a comprehensive understanding of grammar, facts, and logic over time. Note: This is an AI-generated study resource created via NotebookLM based on the Stanford CME295 curriculum and personal study notes.
Embed this episode
No transcript yet.