Tokenizers now and in the future
from Pretrained Hosted by Pierce Freeman & Richard Diehl Martinez
History of tokenization going back to 70s language modeling, modern tokenization approaches with BPE, and the future of token-free bitestreams
Embed this episode
No transcript yet.
Frequently asked questions
How long is this episode of Pretrained?
This episode is 47 minutes long.
When was this Pretrained episode published?
This episode was published on August 14, 2025.
Can I download this Pretrained episode?
Yes. Use the download control on the episode player to save the publisher-provided media file.