#60 - How to input text into your model? Understanding tokenizers.

from Life with AI · host Filipe Lauar

Hello everyone, in this episode I explain how tokenizers work. They are basically what enables us to input the text into a NLP algorithm like BERT or GPT. In the episode I explain 3 types of tokenizers, word based, character based and sub-word based representation. Instagram: https://www.instagram.com/podcast.lifewithai/ Linkedin: https://www.linkedin.com/company/life-with-ai Huuging Face blog about tokenizers: https://huggingface.co/docs/transformers/tokenizer_summary

Episode metadata supplied by the publisher feed · Published Dec 1, 2022

Embed this episode

Attribution link and audio player

Ready to play

#60 - How to input text into your model? Understanding tokenizers.

0:00 14:58

1×

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

No similar podcasts found.

Frequently Asked Questions

How long is this episode of Life with AI?

This episode is 14 minutes long.

When was this Life with AI episode published?

This episode was published on December 1, 2022.

Can I download this Life with AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.

URL copied to clipboard!