# Prompt versions and regression tests interview questions

> 10 prompt versions and regression tests interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/regression-testing
Last updated: 2026-10-23

10 questions: 2 junior, 5 mid-level and 3 senior. Taught in [Versioning Prompts and Catching Regressions](https://llmdojo.dev/learn/prompt-context-engineering/prompt-versioning).

## [Why version prompts, and what should be logged with every output?](https://llmdojo.dev/interview/q/why-version-prompts-and-what-should-be-logged-with-every-output)

Level: Junior

A prompt **shapes behaviour like code**, so a small edit can change outputs invisibly. **Give every version an id**, and log the prompt version, model snapshot and sampling settings with each output, so you can split logs by version and compare. A prompt catalog lets apps sharing a prompt each **pin a version**.

## [A task returns JSON with a label. Which scorer fits it, and why not exact match or a contains check?](https://llmdojo.dev/interview/q/a-task-returns-json-with-a-label-which-scorer-fits-it-and)

Level: Junior

**JSON valid plus label match** on the parsed field. Exact match fails a correct `{"label": "billing"}` because of the JSON around the word. Contains wrongly passes `{"label": "bug", "reason": "not billing"}`. Parsing, then **comparing one field**, catches unparseable output and wrong labels separately.

## [Prompt B passes 9 of 12 test cases; prompt A passed 8. Do you ship B?](https://llmdojo.dev/interview/q/prompt-b-passes-9-of-12-test-cases-prompt-a-passed-8)

Level: Mid-level

**Not on the totals**. In the lesson's illustrative bench, five cases moved: B **fixed three and broke two**, sending an invoice case to account and growing a reason past 120 characters.

**Diagram: Version B: +1 case overall, five cases moved**

1. The totals say B is one case better.
2. B's new rules fixed three cases: the refund, the password reset and the fenced JSON.
3. The same rules broke two that used to pass. 'Profile details are account' pulled in the invoice, and nothing kept the reason short.
4. And one case fails under both. The average hides all of this: five cases moved for a net change of one.

Illustrative outputs in lib/labs/regression.ts (not from a model); counts pinned in lib/labs/regression.test.ts.

An instruction is read for every input, so each break is a trade to review. And with five moved, 3 to 2 gives a sign test p = 1: **no evidence B is better**.

## [On a 20-case suite, a new prompt scores 5 points higher. What can you honestly report?](https://llmdojo.dev/interview/q/on-a-20-case-suite-a-new-prompt-scores-5-points-higher)

Level: Mid-level

That **one more case passed**. On 20 cases one case is **5 points**; even 5 fixes and no breaks gives a sign-test p of 0.0625. The book's guide from OpenAI: about 10 samples to detect a 30% difference with 95% confidence, about 100 for 10%. Read the moved cases, then **grow the set**, or bootstrap it to see how much the score moves.

## [Design the regression gate for a production prompt. What triggers a run, what blocks a merge, and what about the model changing?](https://llmdojo.dev/interview/q/design-the-regression-gate-for-a-production-prompt-what-triggers-a-run)

Level: Senior

Anything that can change an output or its judging triggers it: prompt edits, template updates, sampling settings, scorers, a new model snapshot. A **hash of the prompt text** catches silent edits. **Block on broken cases**, not just the average, and store every run's outputs with their version. The model can change behind the same API: Chen et al. report GPT-4 fell from 84% to 51% on prime versus composite numbers between March and June 2023. **Pin snapshots** or rerun on a schedule, and add every production failure to the set.

## [How do you build a golden dataset, and use it for regression testing?](https://llmdojo.dev/interview/q/how-do-you-build-a-golden-dataset-and-use-it-for-regression)

Level: Mid-level

Start from **real traffic**: typical cases, every bug you've fixed, and the edge cases you fear. Give each an expected answer or a checkable property, reviewed by someone who knows the domain.

**Diagram: Version B: +1 case overall, five cases moved**

1. The totals say B is one case better.
2. B's new rules fixed three cases: the refund, the password reset and the fenced JSON.
3. The same rules broke two that used to pass. 'Profile details are account' pulled in the invoice, and nothing kept the reason short.
4. And one case fails under both. The average hides all of this: five cases moved for a net change of one.

Illustrative outputs in lib/labs/regression.ts (not from a model); counts pinned in lib/labs/regression.test.ts.

Run the set on **every prompt or model change**, compare case by case, and block merges that break a case that used to pass. Keep growing it: each **production failure becomes a test**.

## [Design a system that validates the quality of an LLM's outputs.](https://llmdojo.dev/interview/q/design-a-system-that-validates-the-quality-of-an-llm-s-outputs)

Level: Senior

Two loops. **Offline**: a versioned eval set scored by cheap deterministic checks first, like schema, required facts and banned content, then an **LLM judge** calibrated against human labels, gating every change. **Online**: the same checks on live outputs, sampled judge scores, and user signals such as retries, sliced by task. Failures **flow back into the eval set**, and humans review a sample so the judge doesn't drift.

## [Design the evaluation and improvement loop for an LLM application already in production.](https://llmdojo.dev/interview/q/design-the-evaluation-and-improvement-loop-for-an-llm-application-already-in)

Level: Senior

Start from what success means for the product, then build an **eval set from real traffic**: sampled, labelled, covering known failure types. Score with code checks where possible and **judge models validated against human labels** elsewhere.

**Diagram: Run the suite on every change**

1. Anything that can change the output counts as a change, not only edits to the prompt text.
2. Run the whole fixed suite automatically, the way unit tests run on code.
3. Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
4. When something slips through to users, add it as a case so it can't silently return.

Every prompt, model or retrieval change runs the suite as a **gate** before release. In production, log inputs and outputs, collect user feedback, and canary or A/B test changes. Triage failures into categories, fix the biggest one, and add its examples to the eval set, so each round makes the next measurement sharper.

## [Build a system to manage prompts and model behaviour in production, then defend its versioning and rollout trade-offs.](https://llmdojo.dev/interview/q/build-a-system-to-manage-prompts-and-model-behaviour-in-production-then)

Level: Mid-level

Treat prompts as **versioned config**: a registry stores each prompt's template, model, parameters and output schema as an immutable version that the app references by ID. Every change runs a **regression suite** of golden cases and judge checks before promotion.

**Diagram: Run the suite on every change**

1. Anything that can change the output counts as a change, not only edits to the prompt text.
2. Run the whole fixed suite automatically, the way unit tests run on code.
3. Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
4. When something slips through to users, add it as a case so it can't silently return.

Roll out like a feature flag, with instant rollback. Log the prompt version and model with every output, so production quality can be **sliced by version**.

## [Design a platform that lets teams evaluate their AI agents and see how changing the system prompt or the model affects quality.](https://llmdojo.dev/interview/q/design-a-platform-that-lets-teams-evaluate-their-ai-agents-and-see)

Level: Mid-level

Treat each configuration, model plus system prompt plus tools plus settings, as a **versioned artifact**. Keep scenario suites of recorded and synthetic conversations, with a **simulated user** for multi-turn tests and expected outcomes.

**Diagram: Run the suite on every change**

1. Anything that can change the output counts as a change, not only edits to the prompt text.
2. Run the whole fixed suite automatically, the way unit tests run on code.
3. Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
4. When something slips through to users, add it as a case so it can't silently return.

Run both versions on the same scenarios several times, score with deterministic checks and calibrated judges, and report **paired differences** with confidence intervals per slice. Block on regressions in critical cases, and refresh suites from production traffic.

## Related
- [All interview questions](https://llmdojo.dev/interview)