Mid-level interview questionAsked at Google · interview write-up
What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?
Model answer
Perplexity is exp of the mean negative log-likelihood per token, the model's effective branching factor on held-out text. Being per token, it isn't comparable across tokenizers: longer tokens mean fewer, harder predictions. Normalise to bits per byte to compare. It also measures fit to a text distribution, not helpfulness, correctness or instruction following, so it says little about a chat model's quality, and it shifts with the eval set.
Practise the rest
11 Benchmarking questions, 492 in all, each with a model answer. Free.
More from Google interview questions.
More Benchmarking interview questions
- What does a fair inference benchmark hold fixed?
- An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?
- A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?
- A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
- Design optimised inference pipelines, and tell me how you would benchmark them.