Topic 8

Hands-on and wrap-up

Two places where tokenization shows up in everyday engineering:

1. Budgeting a prompt. Before calling a model, count tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught with that model's tokenizerTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught to check the prompt fits the context windowContext windowThe maximum number of tokens a model can process at once: prompt and output together.Where this is taught, leave room for the output, and estimate cost. Counting locally is cheap and avoids a failed request.
2. Chunking documents for retrieval. When you split documents into chunks for a retrieval system, you can measure chunk size in tokens of your generation model's tokenizer. That makes it easy to guarantee chunks fit the context window. The cost, as AI Engineering points out, is that switching to a model with a different tokenizer means re-chunking and re-indexing your data.

Try it right here

These are the real tokenizers GPT models use, running in your browser.

1. Start with Guess the count: four rounds where you commit to a number before you see the split.
2. Then switch to Explore and try your own text: your name in normal case and in capitals, a sentence in another language you know, a long number with and without commas, an indented code block. Watch how the three tokenizers cut the same text differently.

Try it on your machine

Install tiktoken (pip install tiktoken) and run:

import tiktoken

text = "Tokenizers turn text into integers."
for name in ["gpt2", "cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(name)
    ids = enc.encode(text)
    pieces = [enc.decode([i]) for i in ids]
    print(f"{name}: {len(ids)} tokens")
    print("  ids:   ", ids)
    print("  pieces:", pieces)
    assert enc.decode(ids) == text  # round-trips losslessly

Output with tiktoken 0.14.0 (the first run downloads each encoding file):

gpt2: 7 tokens
  ids:    [30642, 11341, 1210, 2420, 656, 37014, 13]
  pieces: ['Token', 'izers', ' turn', ' text', ' into', ' integers', '.']
cl100k_base: 7 tokens
  ids:    [3404, 12509, 2543, 1495, 1139, 26864, 13]
  pieces: ['Token', 'izers', ' turn', ' text', ' into', ' integers', '.']
o200k_base: 7 tokens
  ids:    [4421, 24223, 3716, 2201, 1511, 54912, 13]
  pieces: ['Token', 'izers', ' turn', ' text', ' into', ' integers', '.']

Probe the quirks

Before running this, write down your guess for each count.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
for s in ["strawberry", " strawberry", "2026", "1234567", "HELLO"]:
    ids = enc.encode(s)
    print(repr(s), len(ids), [enc.decode([i]) for i in ids])

'strawberry' 3 ['st', 'raw', 'berry']
' strawberry' 1 [' strawberry']
'2026' 2 ['202', '6']
'1234567' 3 ['123', '456', '7']
'HELLO' 2 ['HEL', 'LO']

Now try your own text: a paragraph in another language you know, a JSON object, a block of indented code. Guess first, then check.

Running example

One sentence follows you through this whole module. Here is what it looks like after this lesson: just token IDsToken IDThe integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.Where this is taught.

Summary

• Models consume integer token IDs, not text. A tokenizer maps text → tokens → IDs and back, and each ID later selects a row in the model's embedding tableEmbedding tableThe learned matrix with one row per vocabulary entry. Embedding a token ID means reading its row.Where this is taught.
• SubwordSubwordA token that can be a whole word or a fragment of one. Common words get one token; rare words are built from several.Where this is taught vocabularies trade off vocabularyVocabularyThe fixed list of every token a tokenizer knows. Its size is a design choice made before training.Where this is taught size against sequence length: common words become one token, rare words are assembled from smaller pieces, and with a byte-level base nothing is unrepresentable.
• BPEByte pair encoding (BPE)A way to learn a subword vocabulary: start from single symbols and repeatedly merge the most frequent adjacent pair. Encoding replays the learned merges in order.Where this is taught learns an ordered list of merges by repeatedly fusing the most frequent adjacent pair, then encodes new text by replaying those merges in order.
• Token counts depend on language, code, numbers, whitespace, capitalization and the tokenizer. Count with the real tokenizer; do not estimate from words.
• Tokenizers are tied to their models. Never mix them, and never let untrusted text become special tokensSpecial tokenA vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.Where this is taught.

Next

Next lesson: embeddingsEmbeddingA vector of numbers that represents a token or a text, arranged so that similar meanings end up close together.Where this is taught, where each token ID becomes a vector and similarity between meanings becomes geometry. The tokenizer lab above stays here whenever you want to check a token count.