“are very short. There's some long context documents, but…” — Latent Space: The AI Engineer Podcast at 30:43
“are very short. There's some long context documents, but the vast majority of them, I think last time I measured it was something like 90% are of context length 2K or below. So if you're doing really long context training runs on that data, typically how people do it, they call it packing, right? They just sort of…”
0:00
32:59