# Tokenization interview questions

> 11 tokenization interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/tokenization
Last updated: 2026-10-23

11 questions: 4 junior, 5 mid-level and 2 senior. Taught in [Tokenization: From Text to Token IDs](https://llmdojo.dev/learn/model-foundations/tokenization).

## [When you send a prompt to a language model, what does the model actually receive?](https://llmdojo.dev/interview/q/when-you-send-a-prompt-to-a-language-model-what-does-the)

Level: Junior

A list of **integer IDs**, not the string. A tokenizer swaps each piece of text for an ID from a **fixed vocabulary**; the model reads IDs and writes them **one at a time**. With GPT-4o's o200k_base, "Tokenizers turn text into integers." is seven tokens: `Token` + `izers`, and the period on its own.

## [What is a token, and why do most models use subword tokens rather than whole words or single characters?](https://llmdojo.dev/interview/q/what-is-a-token-and-why-do-most-models-use-subword-tokens)

Level: Junior

A token is a piece of text from the tokenizer's fixed vocabulary, stored as an **integer ID**. **Subwords are the compromise**: whole words need a huge vocabulary and miss new words; characters make sequences several times longer. Rare words are built from frequent fragments, and a **byte-level base** like GPT-2's encodes any input.

## [How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?](https://llmdojo.dev/interview/q/how-does-byte-pair-encoding-learn-a-vocabulary-and-how-does-it)

Level: Junior

Training repeatedly merges the **most frequent adjacent pair** of symbols into a new one, giving an ordered merge list. Encoding splits new text into base symbols and **replays the merges in order**, so an unseen word becomes several known pieces. GPT-2's **50,257** entries are 256 byte tokens, 50,000 merges and one end-of-text token.

## [A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?](https://llmdojo.dev/interview/q/a-colleague-estimates-next-month-s-api-bill-by-counting-the-words)

Level: Mid-level

Prices are per token of **one specific tokenizer**, and **words-to-tokens isn't constant**. The rule of about 100 tokens per 75 words is for English prose with GPT-4's tokenizer. With cl100k_base, an English sentence took 6 tokens and a Hindi one saying roughly the same took 21, so the Hindi tickets would be badly underestimated.

**Sample real tickets**, count them with the tokenizer of the model you'll call, and budget for output tokens.

## [You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?](https://llmdojo.dev/interview/q/you-re-moving-a-product-from-a-model-that-uses-cl100k-base)

Level: Senior

Anything that stores tokens, because tokenizer mismatches **fail silently**: decoding IDs with the wrong tokenizer returns **plausible garbage**, not an error. The plan:

- **Cached token IDs**: rebuild them.
- **Chunks sized in tokens**: re-chunk and re-index.
- **Token budgets and cost estimates**: recount; the same text gives different counts.
- **Throughput dashboards**: tokens per second now means a different amount of text, so old and new numbers don't compare.

Keeping the old tokenizer isn't an option: model and tokenizer were **trained together**.

## [A ticket summariser crashes because tiktoken raises ValueError on a ticket that contains the literal end-of-text marker. A teammate's fix is to pass allowed_special='all'. What's your review comment?](https://llmdojo.dev/interview/q/a-ticket-summariser-crashes-because-tiktoken-raises-valueerror-on-a-ticket-that)

Level: Senior

**Block it**: that change *is* the injection. Special tokens like `<|endoftext|>` carry **control meaning**, so with `allowed_special="all"` a user who types the marker can end a turn early or forge a boundary. The ValueError is the defence working.

Encode untrusted text with `encode_ordinary()` or `disallowed_special=()`, so the marker stays **plain characters**; allowed, it becomes ID 199999, a real end-of-text signal. Add special tokens only from your own code. And catching the error to drop the ticket isn't a fix either: it silently loses data.

## [Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?](https://llmdojo.dev/interview/q/compare-the-main-nlp-tokenization-approaches-then-how-would-you-design-a)

Level: Mid-level

Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long. **Subword** methods, BPE, WordPiece and unigram, sit between: frequent words stay whole and rare ones split, so nothing is out of vocabulary.

For the recommender, keep a classic retrieval stage over item and user-history embeddings, and use the LLM to **rerank and explain** a short candidate list, since item IDs make poor tokens and catalogues change daily.

## [How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?](https://llmdojo.dev/interview/q/how-would-you-design-a-subword-tokenizer-for-a-new-llm-and)

Level: Mid-level

Train **byte-level BPE** on a sample of the real mix of languages and code, so nothing is unknown. Vocabulary size trades shorter sequences against bigger embedding and output tables. Add digit and whitespace rules and chat tokens.

**KL-regularised SFT** adds β times the KL divergence from a frozen base model's next-token distributions to the cross-entropy loss. It **limits drift and forgetting**; β sets how far it may move.

## [What is subword tokenization, what does SentencePiece add, and what are the main transformer variants?](https://llmdojo.dev/interview/q/what-is-subword-tokenization-what-does-sentencepiece-add-and-what-are-the)

Level: Junior

**Subword** tokenizers keep common words whole and split rare ones into frequent pieces: no unknown words, short sequences. **SentencePiece** trains on raw text, spaces included, so it needs no language-specific pre-splitting.

Variants: **encoder-only** like BERT for understanding, decoder-only like GPT for generation, and encoder-decoder like T5 for input-to-output tasks such as translation.

## [Implement a byte-pair encoding tokenizer with train, encode and decode, where training keeps merging the most frequent adjacent pair until counts fall below a threshold.](https://llmdojo.dev/interview/q/implement-a-byte-pair-encoding-tokenizer-with-train-encode-and-decode-where)

Level: Mid-level

Work on bytes so any text encodes. Training merges the **most frequent adjacent pair** into a new ID until its count falls below the threshold.

```python
from collections import Counter

def pairs(ids):
    return zip(ids, ids[1:])

def merge(ids, pair, new):
    out, i = [], 0
    while i < len(ids):
        if tuple(ids[i:i + 2]) == pair:
            out.append(new)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

def train(text, min_count):
    ids, merges = list(text.encode()), {}
    while len(ids) > 1:
        pair, n = Counter(pairs(ids)).most_common(1)[0]
        if n < min_count:
            break
        merges[pair] = 256 + len(merges)
        ids = merge(ids, pair, merges[pair])
    return merges

def encode(text, merges):
    ids = list(text.encode())
    while len(ids) > 1:
        cand = [p for p in set(pairs(ids)) if p in merges]
        if not cand:
            break
        p = min(cand, key=merges.get)  # earliest merge
        ids = merge(ids, p, merges[p])
    return ids

def decode(ids, merges):
    vocab = {i: bytes([i]) for i in range(256)}
    for (a, b), i in merges.items():  # in merge order
        vocab[i] = vocab[a] + vocab[b]
    return b"".join(vocab[i] for i in ids).decode(
        "utf-8", errors="replace")
```

Encoding applies merges in **training order**, always the earliest-learned pair present; decoding concatenates byte strings. Interviewers check the round trip, overlapping pairs like aaa, and the stopping rule.

## [Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?](https://llmdojo.dev/interview/q/before-coding-a-bert-keyphrase-span-extraction-task-what-would-you-clarify)

Level: Mid-level

Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings?

Tokenizer: which vocabulary, cased or not. Map character labels to token start and end with the fast tokenizer's **offset mapping**, and decide what happens when a label cuts through a subword. Texts over 512 tokens need **sliding windows**.

Valid spans: end not before start, a maximum length, a no-answer case. Outputs: top spans as **character offsets** with scores.

## Related
- [All interview questions](https://llmdojo.dev/interview)