Junior interview questionAsked by candidates · YouTube
Why version prompts, and what should be logged with every output?
Model answer
A prompt shapes behaviour like code, so a small edit can change outputs invisibly. Give every version an id, and log the prompt version, model snapshot and sampling settings with each output, so you can split logs by version and compare. A prompt catalog lets apps sharing a prompt each pin a version.
Practise the rest
10 Regression tests questions, 492 in all, each with a model answer. Free.
More Regression tests interview questions
- A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?
- Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?
- On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?
- Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?
- How do you build a golden dataset, and use it for regression testing?
- Design a system that validates the quality of an LLM's outputs.