Topic 1

Models do not read text

A language model is a function over numbers. It never sees the string you send it. Before your prompt reaches the model, a separate component called the tokenizerTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught → cuts the text into pieces called tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught → and replaces each piece with an integer, its token IDToken IDThe integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.Where this is taught →. The model reads that list of integers and, when it generates, produces one token ID at a time, which the same tokenizer turns back into text.

The real output

Every value below was produced with tiktoken 0.14.0 on Python 3.14.7, using three OpenAI encodings: gpt2, cl100k_base (used by GPT-4 and GPT-3.5-turbo) and o200k_base (used by GPT-4o).

text:    "Tokenizers turn text into integers."

pieces:  'Token' 'izers' ' turn' ' text' ' into' ' integers' '.'

ids (gpt2):         [30642, 11341, 1210, 2420, 656, 37014, 13]
ids (cl100k_base):  [3404, 12509, 2543, 1495, 1139, 26864, 13]
ids (o200k_base):   [4421, 24223, 3716, 2201, 1511, 54912, 13]

Three things to notice:

• Five words became seven tokens. A rarer word is split into pieces; punctuation is its own token.
• Spaces belong to the token that follows them. The piece is ' turn', with the leading space, not 'turn'. There is no separate token for the space between words.
• The pieces are identical across the three encodings, but the IDs are not. An ID only has meaning inside the vocabularyVocabularyThe fixed list of every token a tokenizer knows. Its size is a design choice made before training.Where this is taught → it came from. ID 13 is . in all three only because these encodings list their 256 single-byte base tokens in the same order; above that range the IDs diverge.

The vocabulary is a fixed design choice made before the model is trained. For the three encodings above, tiktoken reports vocabulary sizes (n_vocab, which includes special tokensSpecial tokenA vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.Where this is taught →) of 50,257, 100,277 and 200,019.