Lesson 1 · 8 topics · 18 min · beginner

Tokenization: From Text to Token IDs

Draft, not yet technically reviewed. Treat its claims as provisional until it is published.

You will be able to

Explain what a token is and trace text to tokens to integer IDs to an embedding lookup.
Explain why LLMs use subword tokenization instead of words or characters, and the vocabulary-size versus sequence-length trade-off.
Train and apply byte pair encoding by hand on a small corpus, distinguishing learning merges from replaying them.
Predict why token counts differ from word counts across languages, code, numbers, whitespace and capitalization.
Avoid common tokenizer mistakes: word-based cost estimates, mixing tokenizers, and letting untrusted text become special tokens.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 2 "Tokens and Embeddings", sections "LLM Tokenization", "How Does the Tokenizer Break Down Text?", "Word Versus Subword Versus Character Versus Byte Tokens", "Comparing Trained LLM Tokenizers", "Tokenizer Properties", "A Language Model Holds Embeddings for the Vocabulary of Its Tokenizer".
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 1 "Introduction to Building AI Applications with Foundation Models", section "Language models"; Ch. 2 "Understanding Foundation Models", section "Multilingual Models"; Ch. 6 "RAG and Agents", section "Chunking strategy"; Ch. 9 "Inference Optimization", section "Throughput and goodput".
Neural Machine Translation of Rare Words with Subword Units · Rico Sennrich et al., 2016
Rico Sennrich, Barry Haddow, Alexandra Birch. ACL 2016 (arXiv:1508.07909). Open source
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Taku Kudo et al., 2018
Taku Kudo, John Richardson. EMNLP 2018 System Demonstrations (arXiv:1808.06226). Open source
tiktoken · OpenAI
OpenAI. Open source
Tokenization algorithms (Transformers documentation) · Hugging Face
Hugging Face. Open source

You will be able to

Explain what a token is and trace text to tokens to integer IDs to an embedding lookup.

Explain why LLMs use subword tokenization instead of words or characters, and the vocabulary-size versus sequence-length trade-off.

Train and apply byte pair encoding by hand on a small corpus, distinguishing learning merges from replaying them.

Predict why token counts differ from word counts across languages, code, numbers, whitespace and capitalization.

Avoid common tokenizer mistakes: word-based cost estimates, mixing tokenizers, and letting untrusted text become special tokens.

Sources

Hands-On Large Language Models · Jay Alammar et al., 2024

Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 2 "Tokens and Embeddings", sections "LLM Tokenization", "How Does the Tokenizer Break Down Text?", "Word Versus Subword Versus Character Versus Byte Tokens", "Comparing Trained LLM Tokenizers", "Tokenizer Properties", "A Language Model Holds Embeddings for the Vocabulary of Its Tokenizer".

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 1 "Introduction to Building AI Applications with Foundation Models", section "Language models"; Ch. 2 "Understanding Foundation Models", section "Multilingual Models"; Ch. 6 "RAG and Agents", section "Chunking strategy"; Ch. 9 "Inference Optimization", section "Throughput and goodput".

Neural Machine Translation of Rare Words with Subword Units · Rico Sennrich et al., 2016

Rico Sennrich, Barry Haddow, Alexandra Birch. ACL 2016 (arXiv:1508.07909). Open source

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Taku Kudo et al., 2018

Taku Kudo, John Richardson. EMNLP 2018 System Demonstrations (arXiv:1808.06226). Open source

tiktoken · OpenAI

OpenAI. Open source

Tokenization algorithms (Transformers documentation) · Hugging Face

Hugging Face. Open source