Senior interview question
How do you choose a scoring method for a new LLM feature?
Model answer
Take the most exact method the task allows.
Scroll sideways to see the whole diagram
If the output can be run and checked, such as code against tests, that's the score to use.
Text version
- If the output can be run and checked, such as code against tests, that's the score to use.
- If not, but there's one short right answer, compare exactly, after normalising case and spacing.
- If many phrasings are right and you have good references, compare by shared words or, better, by meaning.
- If none of those fit, use a judge with a written rubric.
- And check the metric itself: read a few high-scoring outputs and make sure they're actually good.
If the output can be run and checked, use functional correctness. One short right answer: exact match after normalising. Many valid phrasings and good references: lexical or semantic similarity. Otherwise, a judge with a rubric. Then check the metric itself by reading outputs it scores high and low.
Practise the rest
8 Reference scoring questions, 492 in all, each with a model answer. Free.
More Reference scoring interview questions
- What's the difference between exact and subjective evaluation?
- What is pass@k?
- Why can BLEU or ROUGE mislead you when evaluating a model?
- When would you use semantic similarity instead of word overlap, and what does it depend on?
- Can perplexity tell you which of two chat models is better?
- Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.