Fine-Tuning a Trillion-Parameter Model at Home
from My Weird Prompts
Daniel fine-tuned a speech model on his own mixed Hebrew-and-English recordings for the price of a coffee. So what happens if you point that same move at an Opus 5.5-class frontier model — raw, full precision, no practical reason at all? This episode walks through the arithmetic: the sixteen bytes per parameter that Adam demands, why a one-trillion-parameter model needs roughly 200 H100s just to hold its training state, and why mixture-of-experts saves compute but not memory. Along the way: what Modal's eight-GPU ceiling actually buys you, why a ten-billion-token fine-tune costs about $166,000 in pure compute, and what 1,024 GPUs and 1.8 megawatts of power look like as a physical room. Episode 6217 — myweirdprompts.com/n/6217 (permanent short link: myweirdprompts.com/724500) Listen on Android: MWP FM Player — myweirdprompts.com/apk
Embed this episode
Chapters
No transcript yet.