Topic 2
Why subwords
Any tokenizer has to answer one question: what is the unit? The three obvious answers each fail differently.
| Unit | Vocabulary size | Sequence length | Unseen words |
|---|---|---|---|
| Whole words | Huge (every form of every word) | Short | Cannot be represented; fall back to an "unknown" token |
| Characters | Tiny | Long (one token per letter) | Always representable |
| Subwords | Moderate, chosen by the designer | In between | Representable by splitting into smaller known pieces |
Word-level vocabularies explode: "connect", "connects", "connected", "connecting" and "connection" each need their own entry, and a word coined after training (a new product name, a typo) has no entry at all. Every vocabulary entry also costs a row in the model's embedding table, so a giant vocabulary is a memory cost, not only a lookup cost.
Character-level tokenization solves the unknown-word problem, but each token carries very little meaning, and sequences get several times longer. Since models have a fixed context window and pay compute per token, longer sequences mean less text fits and every request costs more.
Subword tokenization takes the middle path. Frequent words become a single token, while rare words are assembled from frequent fragments. On the previous page, the common word text was one token while the rarer "Tokenizers" was two. As long as the base pieces cover every possible input (byte-level tokenizers guarantee this, as you will see), nothing is truly unknown: in the worst case a word is spelled out from very small pieces. Older character-based tokenizers could still hit an "unknown" token for characters they never saw, such as emoji.
Whole words
Characters
Subwords (o200k_base)
Three ways to cut one word. Word-level vocabularies can't represent words they never saw; character-level ones can, but make sequences long. Subwords keep common words whole and spell out rare ones from pieces.
Subword split measured with tiktoken 0.14.0 (o200k_base). The [UNK] row is illustrative: it shows what a word-level vocabulary does with a missing word.
The dial
Vocabulary size is a dial the tokenizer designer sets:
- Larger vocabulary: shorter sequences (cheaper, more text per context window), but a bigger embedding table and fewer training examples for each rare token.
- Smaller vocabulary: a smaller table, but longer sequences.
That is why vocabulary size is a deliberate design decision and differs from model to model.
Quick check