Senior interview question
Can perplexity tell you which of two chat models is better?
Model answer
Not on its own. Perplexity measures how predictable a text is to a model. It's a rough capability proxy, and useful for spotting contamination, when it's suspiciously low on a benchmark, or odd text. But post-training such as SFT and RLHF typically raises perplexity while making models better at tasks, so judge chat models on task metrics.
Practise the rest
8 Reference scoring questions, 492 in all, each with a model answer. Free.
More Reference scoring interview questions
- What's the difference between exact and subjective evaluation?
- What is pass@k?
- Why can BLEU or ROUGE mislead you when evaluating a model?
- When would you use semantic similarity instead of word overlap, and what does it depend on?
- How do you choose a scoring method for a new LLM feature?
- Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.