Junior interview question
A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
Model answer
JSON valid plus label match on the parsed field. Exact match fails a correct {"label": "billing"} because of the JSON around the word. Contains wrongly passes {"label": "bug", "reason": "not billing"}. Parsing, then comparing one field, catches unparseable output and wrong labels separately.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- How do you build a golden dataset, and use it for regression testing?
- Design a system that validates the quality of an LLM's outputs.