Tokenizer
Also called tokenizers
The component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.
Interview questions about Tokenizer
- How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?
Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown.
- Implement a byte-pair encoding tokenizer with train, encode and decode, where training keeps merging the most frequent adjacent pair until counts fall below a threshold.
Work on bytes so any text encodes. Training merges the most frequent adjacent pair into a new ID until its count falls below the threshold.
- Before coding a BERT keyphrase span-extraction task, what would you clarify about the interface, tokenizer offsets, valid spans and the outputs?
Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings? Tokenizer: which vocabulary, cased or not.
- What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?
Perplexity is exp of the mean negative log-likelihood per token, the model's effective branching factor on held-out text.
- When you send a prompt to a language model, what does the model actually receive?
A list of integer IDs, not the string. A tokenizer swaps each piece of text for an ID from a fixed vocabulary; the model reads IDs and writes them one at a…
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID.
- A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?
Prices are per token of one specific tokenizer, and words-to-tokens isn't constant.
- You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?
Anything that stores tokens, because tokenizer mismatches fail silently: decoding IDs with the wrong tokenizer returns plausible garbage, not an error.