Mid-level interview questionAsked by candidates · collected on GitHub
How do you build a golden dataset, and use it for regression testing?
Model answer
Start from real traffic: typical cases, every bug you've fixed, and the edge cases you fear. Give each an expected answer or a checkable property, reviewed by someone who knows the domain.
Scroll sideways to see the whole diagram
The totals say B is one case better.
Text version
- The totals say B is one case better.
- B's new rules fixed three cases: the refund, the password reset and the fenced JSON.
- The same rules broke two that used to pass. 'Profile details are account' pulled in the invoice, and nothing kept the reason short.
- And one case fails under both. The average hides all of this: five cases moved for a net change of one.
Illustrative outputs in lib/labs/regression.ts (not from a model); counts pinned in lib/labs/regression.test.ts.
Run the set on every prompt or model change, compare case by case, and block merges that break a case that used to pass. Keep growing it: each production failure becomes a test.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- Design a system that validates the quality of an LLM's outputs.