Mid-level interview question
A colleague estimates next month's API bill by counting the words in a batch of English and Hindi support tickets and multiplying by the price per token. What's wrong with the estimate, and how would you do it instead?
Model answer
Prices are per token of one specific tokenizer, and words-to-tokens isn't constant. The rule of about 100 tokens per 75 words is for English prose with GPT-4's tokenizer. With cl100k_base, an English sentence took 6 tokens and a Hindi one saying roughly the same took 21, so the Hindi tickets would be badly underestimated.
Sample real tickets, count them with the tokenizer of the model you'll call, and budget for output tokens.
Practise the rest
11 Tokenization questions, 492 in all, each with a model answer. Free.
More Tokenization interview questions
- When you send a prompt to a language model, what does the model actually receive?
- What is a token, and why do most models use subword tokens rather than whole words or single characters?
- How does byte pair encoding learn a vocabulary, and how does it then encode a word it never saw in training?
- You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?
- A ticket summariser crashes because tiktoken raises ValueError on a ticket that contains the literal end-of-text marker. A teammate's fix is to pass allowed_special='all'. What's your review comment?
- Compare the main NLP tokenization approaches. Then, how would you design a recommender that uses an LLM?