“Yeah, speculative decoding, I would say at this point is a…” — Software Engineering Daily at 25:46
“Yeah, speculative decoding, I would say at this point is a pretty well-known concept for many serving stack. It's a set up where you have a small model that try to guess which token the big model would like and just ask the big model, do you like these tokens?”
0:00
53:13