Topic 6

What this breaks in practice

Limits and prices are in tokens

Context windowsContext windowThe maximum number of tokens a model can process at once: prompt and output together.Where this is taught, output limits, API prices and many rate limits are denominated in tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught of a specific model's tokenizerTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught. A 1,000-word document is not a fixed number of tokens: it depends on the language, the content and the tokenizer. Budget with the tokenizer, not a word count.

Different models, different tokenizers

The model and its tokenizer are trained together: each row of the model's embedding tableEmbedding tableThe learned matrix with one row per vocabulary entry. Embedding a token ID means reading its row.Where this is taught belongs to one entry in that tokenizer's vocabularyVocabularyThe fixed list of every token a tokenizer knows. Its size is a design choice made before training.Where this is taught. You cannot swap in a different tokenizer without retraining. Token counts are therefore not comparable across model families either, and a throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught number quoted in "tokens per second" means different amounts of text for different models.

Never mix tokenizers

Token IDsToken IDThe integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.Where this is taught are only meaningful with the vocabulary that produced them. Encoding "Tokenizers turn text into integers." with o200k_base and decoding the resulting IDs with cl100k_base does not raise an error. It silently returns:

That is the dangerous part: a mismatch produces plausible-looking garbage instead of a crash. The same applies to caches of token IDs, token-based chunk boundaries and token-count budgets stored in a database.

Spelling and arithmetic quirks

The model never sees letters or digits unless the tokenizer happens to cut text that finely. In o200k_base, strawberry at the start of a string is three tokens (st, raw, berry) but strawberry after a space is a single token. Asking "how many r's are in strawberry?" asks the model about letters inside a symbol it was never shown letter by letter. Similarly, 12345 + 67890 = 80235 splits into 123 45 + 678 90 = 802 35, so digit columns do not line up between the numbers.

Tokenization is one commonly cited reason models stumble on character-level and arithmetic tasks. It is not the only factor, and models can still learn these tasks, but it explains why the failures look the way they do.

Special tokens

Besides text pieces, every vocabulary reserves special tokensSpecial tokenA vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.Where this is taught with a control meaning rather than a textual one: end of text, beginning of a sequence, padding, chat-role markers, or fill-in-the-middle markers for code completion. In tiktoken, gpt2 has one (<|endoftext|>, ID 50256); cl100k_base adds <|fim_prefix|>, <|fim_middle|>, <|fim_suffix|> and <|endofprompt|>; o200k_base has <|endoftext|> (ID 199999) and <|endofprompt|>. Chat models use special tokens to mark where one speaker's turn ends and the next begins, which is why chat APIs build that framing for you.

The next page shows how special tokens become a security problem.