#60 - How to input text into your model? Understanding tokenizers.

An episode of the Life with AI podcast, hosted by Filipe Lauar, titled "#60 - How to input text into your model? Understanding tokenizers." was published on December 1, 2022 and runs 14 minutes.

December 1, 2022 ·14m · Life with AI

0:00 / 0:00

Summary

Hello everyone, in this episode I explain how tokenizers work. They are basically what enables us to input the text into a NLP algorithm like BERT or GPT. In the episode I explain 3 types of tokenizers, word based, character based and sub-word based representation. Instagram: https://www.instagram.com/podcast.lifewithai/ Linkedin: https://www.linkedin.com/company/life-with-ai Huuging Face blog about tokenizers: https://huggingface.co/docs/transformers/tokenizer_summary

Episode Description

Hello everyone, in this episode I explain how tokenizers work. They are basically what enables us to input the text into a NLP algorithm like BERT or GPT. In the episode I explain 3 types of tokenizers, word based, character based and sub-word based representation.

Instagram: https://www.instagram.com/podcast.lifewithai/

Linkedin: https://www.linkedin.com/company/life-with-ai

Huuging Face blog about tokenizers: https://huggingface.co/docs/transformers/tokenizer_summary

Share this episode

Similar Episodes

Episode 34: Sight & Sound's Josh and Kristen Enck | A Great Awakening

Mar 19, 2026 ·44m

Episode 33: Russ Taff and Steve Taylor | Trailblazers in the Christian Music Industry

Mar 12, 2026 ·38m

Episode 32: Bart Millard & John Michael Finley | “I Can Only Imagine 2”

Feb 9, 2026 ·28m

Episode 31: Tommy Coomes & Ron Strand “A Band Called Love Song”

Sep 11, 2025 ·29m

New Season, Same God

Jan 15, 2025 ·15m

Standing On Business

Jan 15, 2025 ·18m

Similar Podcasts

Life with Rev. Q Quinton Steverson Daily conversations filled with inspirations, insightful thought and in depth prospectives aimed to motivate and challenge you in this journey called life. Ireland Creates with Aisling O' Rourke Aisling O' Rourke Ireland Creates with Aisling O' Rourke is a podcast about Ireland's storytellers. The story behind storytelling. In each episode, Aisling speaks with a different creative about their life, creative practice and what storytelling means to them. Ireland Creates has featured Writers, Photographers, Musicians and Filmmakers to name just a few. If you would like to get involved please check out www.irelandcreates.com or email Aisling on [email protected] Leanne Wood Podcast Leanne Wood With more than twenty five years experience as a political activist on the Welsh left in various roles, including the leadership of Plaid Cymru and Member of the Senedd for the Rhondda, I am now seeking to contribute to political life with a podcast.This new platform aims to give voice to those people and issues we don’t hear enough from or about in the mainstream media. Our politics needs a different conversation. I hope that this podcast will help with that. Confession Session With Life Coach Francis Camia FRANCIS CAMIA This podcast focuses on life's confessions and discussions that will hopefully make an impact to everyone and lead to a better transformation of themselves. We will be talking about purpose, relationships and anything under the sun. It also aims to shed some hope and positivity to all the listeners. On a personal note, you may also send a message or if you want to have a personal catch up or a collaboration. Let's talk!

URL copied to clipboard!