Mid-level interview question
On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
Model answer
That one more case passed. On 20 cases one case is 5 points; even 5 fixes and no breaks gives a sign-test p of 0.0625. The book's guide from OpenAI: about 10 samples to detect a 30% difference with 95% confidence, about 100 for 10%. Read the moved cases, then grow the set, or bootstrap it to see how much the score moves.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- How do you build a golden dataset, and use it for regression testing?
- Design a system that validates the quality of an LLM's outputs.