Mid-level interview questionAsked at Luma AI · interview write-up

Build a system to manage prompts and model behaviour in production, then defend its versioning and rollout trade-offs.

Q: Build a system to manage prompts and model behaviour in production, then defend its versioning and rollout trade-offs.

Treat prompts as versioned config: a registry stores each prompt's template, model, parameters and output schema as an immutable version that the app references by ID. Every change runs a regression suite of golden cases and judge checks before promotion. Roll out like a feature flag, with instant rollback. Log the prompt version and model with every output, so production quality can be sliced by version.

Model answer

Treat prompts as versioned config: a registry stores each prompt's template, model, parameters and output schema as an immutable version that the app references by ID. Every change runs a regression suite of golden cases and judge checks before promotion.

Run the suite on every changeStep 1 / 4

Scroll sideways to see the whole diagram

Anything that can change the output counts as a change, not only edits to the prompt text.

Text version

Anything that can change the output counts as a change, not only edits to the prompt text.
Run the whole fixed suite automatically, the way unit tests run on code.
Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
When something slips through to users, add it as a case so it can't silently return.

Roll out like a feature flag, with instant rollback. Log the prompt version and model with every output, so production quality can be sliced by version.

Practise the rest

10 Regression tests questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Regression tests

Build a system to manage prompts and model behaviour in production, then defend its versioning and rollout trade-offs.

Model answer

Run the suite on every changeStep 1 / 4

Scroll sideways to see the whole diagram

Anything that can change the output counts as a change, not only edits to the prompt text.

Text version

Anything that can change the output counts as a change, not only edits to the prompt text.
Run the whole fixed suite automatically, the way unit tests run on code.
Gate on broken cases, not only on the total: a better average can still hide a broken case you care about.
When something slips through to users, add it as a case so it can't silently return.

Roll out like a feature flag, with instant rollback. Log the prompt version and model with every output, so production quality can be sliced by version.

Practise the rest

10 Regression tests questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Regression tests

Model answer

More Regression tests interview questions

Model answer

More Regression tests interview questions