Mid-level interview question
Why can BLEU or ROUGE mislead you when evaluating a model?
Model answer
They measure how much the output looks like the references, not whether it's right. A correct rewording can score low because no reference matches it, and a wrong answer that changes one number can score near the top. References can also be incomplete or wrong. On HumanEval, OpenAI found BLEU similar for correct and incorrect solutions.
Practise the rest
8 Reference scoring questions, 492 in all, each with a model answer. Free.
More Reference scoring interview questions
- What's the difference between exact and subjective evaluation?
- What is pass@k?
- When would you use semantic similarity instead of word overlap, and what does it depend on?
- How do you choose a scoring method for a new LLM feature?
- Can perplexity tell you which of two chat models is better?
- Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.