Prompt versions and regression tests interview questions
10 questions with model answers
- junior 2
- mid 5
- senior 3
10 prompt versions and regression tests questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 2 junior, 5 mid-level and 3 senior. The lesson Versioning Prompts and Catching Regressions teaches the ideas behind them.Updated
Model answer
A prompt shapes behaviour like code, so a small edit can change outputs invisibly. Give every version an id, and log the prompt version, model snapshot and sampling settings with each output, so you can split logs by version and compare. A prompt catalog lets apps sharing a prompt each pin a version.
Model answer
JSON valid plus label match on the parsed field. Exact match fails a correct {"label": "billing"} because of the JSON around the word. Contains wrongly passes {"label": "bug", "reason": "not billing"}. Parsing, then comparing one field, catches unparseable output and wrong labels separately.
Model answer
Not on the totals. In the lesson's illustrative bench, five cases moved: B fixed three and broke two, sending an invoice case to account and growing a reason past 120 characters.
Scroll sideways to see the whole diagram
The totals say B is one case better.
Text version
- The totals say B is one case better.
- B's new rules fixed three cases: the refund, the password reset and the fenced JSON.
- The same rules broke two that used to pass. 'Profile details are account' pulled in the invoice, and nothing kept the reason short.
- And one case fails under both. The average hides all of this: five cases moved for a net change of one.
Illustrative outputs in lib/labs/regression.ts (not from a model); counts pinned in lib/labs/regression.test.ts.
An instruction is read for every input, so each break is a trade to review. And with five moved, 3 to 2 gives a sign test p = 1: no evidence B is better.
Model answer
That one more case passed. On 20 cases one case is 5 points; even 5 fixes and no breaks gives a sign-test p of 0.0625. The book's guide from OpenAI: about 10 samples to detect a 30% difference with 95% confidence, about 100 for 10%. Read the moved cases, then grow the set, or bootstrap it to see how much the score moves.
Model answer
Anything that can change an output or its judging triggers it: prompt edits, template updates, sampling settings, scorers, a new model snapshot. A hash of the prompt text catches silent edits. Block on broken cases, not just the average, and store every run's outputs with their version. The model can change behind the same API: Chen et al. report GPT-4 fell from 84% to 51% on prime versus composite numbers between March and June 2023. Pin snapshots or rerun on a schedule, and add every production failure to the set.
Model answer
Start from real traffic: typical cases, every bug you've fixed, and the edge cases you fear. Give each an expected answer or a checkable property, reviewed by someone who knows the domain.
Scroll sideways to see the whole diagram
The totals say B is one case better.
Text version
- The totals say B is one case better.
- B's new rules fixed three cases: the refund, the password reset and the fenced JSON.
- The same rules broke two that used to pass. 'Profile details are account' pulled in the invoice, and nothing kept the reason short.
- And one case fails under both. The average hides all of this: five cases moved for a net change of one.
Illustrative outputs in lib/labs/regression.ts (not from a model); counts pinned in lib/labs/regression.test.ts.
Run the set on every prompt or model change, compare case by case, and block merges that break a case that used to pass. Keep growing it: each production failure becomes a test.
Model answer
Two loops. Offline: a versioned eval set scored by cheap deterministic checks first, like schema, required facts and banned content, then an LLM judge calibrated against human labels, gating every change. Online: the same checks on live outputs, sampled judge scores, and user signals such as retries, sliced by task. Failures flow back into the eval set, and humans review a sample so the judge doesn't drift.
Model answer
Start from what success means for the product, then build an eval set from real traffic: sampled, labelled, covering known failure types. Score with code checks where possible and judge models validated against human labels elsewhere.
Scroll sideways to see the whole diagram
Anything that can change the output counts as a change, not only edits to the prompt text.
Text version
- Anything that can change the output counts as a change, not only edits to the prompt text.
- Run the whole fixed suite automatically, the way unit tests run on code.
- Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
- When something slips through to users, add it as a case so it can't silently return.
Every prompt, model or retrieval change runs the suite as a gate before release. In production, log inputs and outputs, collect user feedback, and canary or A/B test changes. Triage failures into categories, fix the biggest one, and add its examples to the eval set, so each round makes the next measurement sharper.
Model answer
Treat prompts as versioned config: a registry stores each prompt's template, model, parameters and output schema as an immutable version that the app references by ID. Every change runs a regression suite of golden cases and judge checks before promotion.
Scroll sideways to see the whole diagram
Anything that can change the output counts as a change, not only edits to the prompt text.
Text version
- Anything that can change the output counts as a change, not only edits to the prompt text.
- Run the whole fixed suite automatically, the way unit tests run on code.
- Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
- When something slips through to users, add it as a case so it can't silently return.
Roll out like a feature flag, with instant rollback. Log the prompt version and model with every output, so production quality can be sliced by version.
Model answer
Treat each configuration, model plus system prompt plus tools plus settings, as a versioned artifact. Keep scenario suites of recorded and synthetic conversations, with a simulated user for multi-turn tests and expected outcomes.
Scroll sideways to see the whole diagram
Anything that can change the output counts as a change, not only edits to the prompt text.
Text version
- Anything that can change the output counts as a change, not only edits to the prompt text.
- Run the whole fixed suite automatically, the way unit tests run on code.
- Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
- When something slips through to users, add it as a case so it can't silently return.
Run both versions on the same scenarios several times, score with deterministic checks and calibrated judges, and report paired differences with confidence intervals per slice. Block on regressions in critical cases, and refresh suites from production traffic.