Mid-level interview questionAsked by candidates · interview write-up
Design a platform that lets teams evaluate their AI agents and see how changing the system prompt or the model affects quality.
Model answer
Treat each configuration, model plus system prompt plus tools plus settings, as a versioned artifact. Keep scenario suites of recorded and synthetic conversations, with a simulated user for multi-turn tests and expected outcomes.
Scroll sideways to see the whole diagram
Anything that can change the output counts as a change, not only edits to the prompt text.
Text version
- Anything that can change the output counts as a change, not only edits to the prompt text.
- Run the whole fixed suite automatically, the way unit tests run on code.
- Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
- When something slips through to users, add it as a case so it can't silently return.
Run both versions on the same scenarios several times, score with deterministic checks and calibrated judges, and report paired differences with confidence intervals per slice. Block on regressions in critical cases, and refresh suites from production traffic.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- How do you build a golden dataset, and use it for regression testing?