Junior interview question
What is pass@k?
Model answer
For code generation: generate k samples per problem; a problem counts as solved if any of them passes all its tests; pass@k is the share of problems solved. It can only rise with k, so always report k: 80% at k = 10 is much weaker than 80% at k = 1.
Practise the rest
8 Reference scoring questions, 492 in all, each with a model answer. Free.
More Reference scoring interview questions
- What's the difference between exact and subjective evaluation?
- Why can BLEU or ROUGE mislead you when evaluating a model?
- When would you use semantic similarity instead of word overlap, and what does it depend on?
- How do you choose a scoring method for a new LLM feature?
- Can perplexity tell you which of two chat models is better?
- Implement AUC-ROC with correct tie handling, a numerically stable softmax, and logistic regression trained with cross-entropy and L2 regularization.