“text for the AI. But the tokens are not actually always…” — Odd Lots at 18:19
“text for the AI. But the tokens are not actually always three quarters of a word. The tokens are like short words like the or A will sort of be a single token. And long words are broken into multiple tokens. It's sort of the conversion between words and tokens depends on how frequent the word”
0:00
1:16:11