Senior interview question
Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
Model answer
Anything that can change an output or its judging triggers it: prompt edits, template updates, sampling settings, scorers, a new model snapshot. A hash of the prompt text catches silent edits. Block on broken cases, not just the average, and store every run's outputs with their version. The model can change behind the same API: Chen et al. report GPT-4 fell from 84% to 51% on prime versus composite numbers between March and June 2023. Pin snapshots or rerun on a schedule, and add every production failure to the set.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- Why version prompts, and what should be logged with every output?
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- How do you build a golden dataset, and use it for regression testing?
- Design a system that validates the quality of an LLM's outputs.