Topic 5

Token counts are not word counts

Because the vocabularyVocabularyThe fixed list of every token a tokenizer knows. Its size is a design choice made before training.Where this is taught is learned from a training corpus, text that looks like that corpus compresses into few tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught and text that does not falls apart into many small ones. All counts on this page come from tiktoken 0.14.0.

Language

The newer o200k_base encoding, with a vocabulary twice as large, brings Hindi down to 5. Chip Huyen's AI Engineering reports the same pattern at scale: in a benchmark of GPT-4's tokenizerTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught on a multilingual dataset, Hindi and Burmese needed several times more tokens than English to say the same thing, which makes them proportionally slower and more expensive.

(A word count is not even defined for Japanese here: the sentence has no spaces, so splitting on whitespace returns one "word".)

Numbers

In cl100k_base and o200k_base, the pre-tokenization pattern (visible in tiktoken's source) caps digit runs at three, and every 1- to 3-digit string is a single token. GPT-2's pattern takes the whole digit run and lets BPEByte pair encoding (BPE)A way to learn a subword vocabulary: start from single symbols and repeatedly merge the most frequent adjacent pair. Encoding replays the learned merges in order.Where this is taught split it however its merges dictate.

Whitespace and code

gpt2 spends one token per extra space, which makes indented code expensive. The later encodings have tokens for runs of whitespace, a choice that matters for code where indentation is everywhere.

Capitalization and spacing

hello, hello, Hello and Hello are four different single tokens with four unrelated IDs (in o200k_base: 24912, 40617, 13225, 32949). HELLO is two tokens, HEL and LO. To the model, these are different symbols whose relationship it has to learn from data.

A rough rule, used carefully

For ordinary English prose, a token is a bit shorter than a word: AI Engineering gives roughly 100 tokens per 75 words for GPT-4, and the tiktoken README says a token averages about 4 bytes. Those averages break down for non-English text, code, numbers, tables and JSON. When the number matters, count with the real tokenizer.