Tokenization interview questions

A list of integer IDs, not the string. A tokenizer swaps each piece of text for an ID from a fixed vocabulary; the model reads IDs and writes them one at a time. With GPT-4o's o200k_base, "Tokenizers turn text into integers." is seven tokens: Token + izers, and the period on its own.

A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID. Subwords are the compromise: whole words need a huge vocabulary and miss new words; characters make sequences several times longer. Rare words are built from frequent fragments, and a byte-level base like GPT-2's encodes any input.

Training repeatedly merges the most frequent adjacent pair of symbols into a new one, giving an ordered merge list. Encoding splits new text into base symbols and replays the merges in order, so an unseen word becomes several known pieces. GPT-2's 50,257 entries are 256 byte tokens, 50,000 merges and one end-of-text token.

Prices are per token of one specific tokenizer, and words-to-tokens isn't constant. The rule of about 100 tokens per 75 words is for English prose with GPT-4's tokenizer. With cl100k_base, an English sentence took 6 tokens and a Hindi one saying roughly the same took 21, so the Hindi tickets would be badly underestimated.

Sample real tickets, count them with the tokenizer of the model you'll call, and budget for output tokens.

Anything that stores tokens, because tokenizer mismatches fail silently: decoding IDs with the wrong tokenizer returns plausible garbage, not an error. The plan:

• Cached token IDs: rebuild them.
• Chunks sized in tokens: re-chunk and re-index.
• Token budgets and cost estimates: recount; the same text gives different counts.
• Throughput dashboards: tokens per second now means a different amount of text, so old and new numbers don't compare.

Keeping the old tokenizer isn't an option: model and tokenizer were trained together.

Block it: that change is the injection. Special tokens like <|endoftext|> carry control meaning, so with allowed_special="all" a user who types the marker can end a turn early or forge a boundary. The ValueError is the defence working.

Encode untrusted text with encode_ordinary() or disallowed_special=(), so the marker stays plain characters; allowed, it becomes ID 199999, a real end-of-text signal. Add special tokens only from your own code. And catching the error to drop the ticket isn't a fix either: it silently loses data.

Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long. Subword methods, BPE, WordPiece and unigram, sit between: frequent words stay whole and rare ones split, so nothing is out of vocabulary.

For the recommender, keep a classic retrieval stage over item and user-history embeddings, and use the LLM to rerank and explain a short candidate list, since item IDs make poor tokens and catalogues change daily.

Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown. Vocabulary size trades shorter sequences against bigger embedding and output tables. Add digit and whitespace rules and chat tokens.

KL-regularised SFT adds β times the KL divergence from a frozen base model's next-token distributions to the cross-entropy loss. It limits drift and forgetting; β sets how far it may move.

Subword tokenizers keep common words whole and split rare ones into frequent pieces: no unknown words, short sequences. SentencePiece trains on raw text, spaces included, so it needs no language-specific pre-splitting.

Variants: encoder-only like BERT for understanding, decoder-only like GPT for generation, and encoder-decoder like T5 for input-to-output tasks such as translation.

Work on bytes so any text encodes. Training merges the most frequent adjacent pair into a new ID until its count falls below the threshold.

from collections import Counter

def pairs(ids):
    return zip(ids, ids[1:])

def merge(ids, pair, new):
    out, i = [], 0
    while i < len(ids):
        if tuple(ids[i:i + 2]) == pair:
            out.append(new)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

def train(text, min_count):
    ids, merges = list(text.encode()), {}
    while len(ids) > 1:
        pair, n = Counter(pairs(ids)).most_common(1)[0]
        if n < min_count:
            break
        merges[pair] = 256 + len(merges)
        ids = merge(ids, pair, merges[pair])
    return merges

def encode(text, merges):
    ids = list(text.encode())
    while len(ids) > 1:
        cand = [p for p in set(pairs(ids)) if p in merges]
        if not cand:
            break
        p = min(cand, key=merges.get)  # earliest merge
        ids = merge(ids, p, merges[p])
    return ids

def decode(ids, merges):
    vocab = {i: bytes([i]) for i in range(256)}
    for (a, b), i in merges.items():  # in merge order
        vocab[i] = vocab[a] + vocab[b]
    return b"".join(vocab[i] for i in ids).decode(
        "utf-8", errors="replace")

Encoding applies merges in training order, always the earliest-learned pair present; decoding concatenates byte strings. Interviewers check the round trip, overlapping pairs like aaa, and the stopping rule.

Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings?

Tokenizer: which vocabulary, cased or not. Map character labels to token start and end with the fast tokenizer's offset mapping, and decide what happens when a label cuts through a subword. Texts over 512 tokens need sliding windows.

Valid spans: end not before start, a maximum length, a no-answer case. Outputs: top spans as character offsets with scores.