Topic 4

Encoding new text

Training produced an ordered list of merges: a+t, h+at, c+at, cat+s. To encode a word, split it into base symbols and apply those merges in the order they were learned.

• "hats": h a t s → (a+t) h at s → (h+at) hat s. Result: hat, s.
• "cast": no merge rule matches any adjacent pair, so it stays as c, a, s, t.

This is the whole mechanism behind "unknown words get split into known pieces." Try other words below. The stepper starts with the four merges already learned; press Undo to remove merges and watch the same word encode differently.

What real tokenizers add

• Byte-level base vocabularyVocabularyThe fixed list of every token a tokenizer knows. Its size is a design choice made before training.Where this is taught. Instead of letters, GPT-2-style tokenizersTokenizerThe component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.Where this is taught start from the 256 possible byte values, so any input, in any language or containing any emoji, can be encoded without an "unknown" tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught. The Hugging Face documentation describes GPT-2's 50,257-entry vocabulary as 256 byte tokens, 50,000 merges and one end-of-text token.
• Pre-tokenization. Before BPEByte pair encoding (BPE)A way to learn a subword vocabulary: start from single symbols and repeatedly merge the most frequent adjacent pair. Encoding replays the learned merges in order.Where this is taught runs, a regular expression splits the text into chunks (roughly: words with their leading space, runs of digits, punctuation, whitespace), and merges never cross chunk boundaries. This is why a space attaches to the following word.
• Other algorithms. WordPiece (used by BERT) and Unigram (used by T5) build vocabularies differently, and the SentencePiece library trains BPE or Unigram directly on raw text without assuming spaces separate words. The output is the same kind of thing: a vocabulary and a fixed procedure for splitting text into it.