Skip to content

“Finally, the model also makes use of RMSNorm with…” — Y Combinator Startup Podcast at 1:20

“Finally, the model also makes use of RMSNorm with pre-normalization, a normalization method that scales inputs by their root mean square for more stable training. One standard capability of the model is its 131,000 token context window, which it achieves by applying yarn scaling during pre-training rather than as an…”

0:00 12:31