Senior interview questionAsked at Microsoft · interview write-up
Design the evaluation and improvement loop for an LLM application already in production.
Model answer
Start from what success means for the product, then build an eval set from real traffic: sampled, labelled, covering known failure types. Score with code checks where possible and judge models validated against human labels elsewhere.
Scroll sideways to see the whole diagram
Anything that can change the output counts as a change, not only edits to the prompt text.
Text version
- Anything that can change the output counts as a change, not only edits to the prompt text.
- Run the whole fixed suite automatically, the way unit tests run on code.
- Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
- When something slips through to users, add it as a case so it can't silently return.
Every prompt, model or retrieval change runs the suite as a gate before release. In production, log inputs and outputs, collect user feedback, and canary or A/B test changes. Triage failures into categories, fix the biggest one, and add its examples to the eval set, so each round makes the next measurement sharper.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More from Microsoft interview questions.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- How do you build a golden dataset, and use it for regression testing?