Test set
Also called test sets, evaluation set, evaluation sets
A fixed collection of inputs with known expected answers, run unchanged against every version so their results can be compared case by case.
Interview questions about Test set
- For a medical conversational AI, how would you source and split data, choose between prompting, fine-tuning and RAG, and evaluate for hallucination and unsafe advice?
Use vetted clinical guidelines and de-identified, consented conversations, split by patient so nothing leaks into the test set.
- Your few-shot sentiment prompt lists six positive examples, then two negative ones at the very end. What's the risk, and what would you do?
The model may lean towards the label it saw last. Zhao et al.
- Version B has a higher pass rate than version A on last week's production logs. Do you ship B?
Not on that evidence. Each version served different traffic: if B got the easy self-serve tickets, its overall rate is flattered by its mix.
- What is benchmark contamination, and how would you detect it?
A model scoring well because it saw the test in training, usually by accident through web scraping.
- A reranker fixes most of your misses but still prefers “safe for children over three” to “not safe for children under three”. What do you conclude?
A reranker is a better ranker, not a reasoner. On the course's data it ranked the right article first for 5 of 6 queries, but scored the reassuring toy…
- Anthropic reports large gains from contextual retrieval. What exactly did they measure, and what does it cost?
1 − recall@20: the share of relevant chunks missing from the top 20.
- How do you set a difficulty router's threshold?
Trace its cost–accuracy curve on your evaluation set and pick the cheapest point meeting your quality bar.
- Metrics are stable, yet you suspect drift. What drifts in an AI system?
- Prompts: a template changes, someone fixes a typo. - Users: phrasing and topics shift; the book's example is shrinking response lengths as users ask for…