Mid-level interview questionAsked at Amazon · interview write-up
How would you design a subword tokenizer for a new LLM? And separately, what is KL-regularised supervised fine-tuning, and why use it?
Model answer
Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown. Vocabulary size trades shorter sequences against bigger embedding and output tables. Add digit and whitespace rules and chat tokens.
KL-regularised SFT adds β times the KL divergence from a frozen base model's next-token distributions to the cross-entropy loss. It limits drift and forgetting; β sets how far it may move.
Practise the rest
11 Tokenization questions, 492 in all, each with a model answer. Free.
More from Amazon interview questions.
More Tokenization interview questions
- When you send a prompt to a language model, what does the model actually receive?
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
- How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?
- A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?
- You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?
- A ticket summariser crashes because tiktoken raises ValueError on a ticket that contains the literal end-of-text marker. A teammate's fix is to pass allowed_special='all'. What's your review comment?