“to decide which new users get their caches loaded in. You…” — Latent Space: The AI Engineer Podcast at 6:27
“to decide which new users get their caches loaded in. You want to pick ones that fit, but not that are too close to the limit because then you're going to have to rotate them out pretty soon, but not ones that are too far from the limit because having a ton of extra memory is very wasteful. It's just a really…”
0:00
32:59