“small-ish, but real meaningful scale, 3 billion parameter…” — Latent Space: The AI Engineer Podcast at 15:45
“small-ish, but real meaningful scale, 3 billion parameter coding model. And this is the baseline level of loss that it achieves on like a very standard, they call it the stack dataset, specialized to Python. And what happens when you switch it from flash attention to power retention? Well, you get from this blue loss…”
0:00
32:59