Lesson 5 · 8 topics · 22 min · intermediate

Versioning Prompts and Catching Regressions

You will be able to

Keep prompts separate from code, give each version an id, and log the prompt version, model and settings with every output.
Build a fixed test set of cases with expected answers, and explain why both the inputs and the answers must stay fixed between runs.
Choose scorers (exact match, label match, JSON valid, contains, doesn't contain, length limit) that fit a task's output, and explain how contains can pass a wrong answer.
Compare two prompt versions case by case (fixed, broke, still passing, still failing) instead of by their average.
Explain why a prompt change aimed at a few cases can break others.
Compute how much one case is worth on a small test set and an exact sign test on the cases that moved, and report only the confidence that supports.
Run the suite on every change, including template, settings and model changes, and treat a new model snapshot as a new version.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology": "Exact Evaluation" > "Similarity Measurements Against Reference Data" > "Exact match". Ch. 4 "Evaluate AI Systems": "Model Selection" > "Model Build Versus Buy" > "Open source models versus model APIs" > "Control, access, and transparency"; "Design Your Evaluation Pipeline" > "Step 3. Define Evaluation Methods and Data". Ch. 5 "Prompt Engineering": "Introduction to Prompting"; "Prompt Engineering Best Practices" > "Iterate on Your Prompts"; "Evaluate Prompt Engineering Tools"; "Organize and Version Prompts". Ch. 10 "AI Engineering Architecture and User Feedback": "Monitoring and Observability" > "Drift detection".
Hands-On Large Language Models · Jay Alammar & Maarten Grootendorst, 2024
Jay Alammar & Maarten Grootendorst. O'Reilly. Used: Ch. 6 "Prompt Engineering": "Output Verification".
How is ChatGPT's behavior changing over time? · Lingjiao Chen et al., 2023
Lingjiao Chen, Matei Zaharia & James Zou. arXiv:2307.09009. Used: Abstract (fetched 2026-10-21). Open source

You will be able to

Keep prompts separate from code, give each version an id, and log the prompt version, model and settings with every output.

Build a fixed test set of cases with expected answers, and explain why both the inputs and the answers must stay fixed between runs.

Choose scorers (exact match, label match, JSON valid, contains, doesn't contain, length limit) that fit a task's output, and explain how contains can pass a wrong answer.

Compare two prompt versions case by case (fixed, broke, still passing, still failing) instead of by their average.

Explain why a prompt change aimed at a few cases can break others.

Compute how much one case is worth on a small test set and an exact sign test on the cases that moved, and report only the confidence that supports.

Run the suite on every change, including template, settings and model changes, and treat a new model snapshot as a new version.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology": "Exact Evaluation" > "Similarity Measurements Against Reference Data" > "Exact match". Ch. 4 "Evaluate AI Systems": "Model Selection" > "Model Build Versus Buy" > "Open source models versus model APIs" > "Control, access, and transparency"; "Design Your Evaluation Pipeline" > "Step 3. Define Evaluation Methods and Data". Ch. 5 "Prompt Engineering": "Introduction to Prompting"; "Prompt Engineering Best Practices" > "Iterate on Your Prompts"; "Evaluate Prompt Engineering Tools"; "Organize and Version Prompts". Ch. 10 "AI Engineering Architecture and User Feedback": "Monitoring and Observability" > "Drift detection".

Hands-On Large Language Models · Jay Alammar & Maarten Grootendorst, 2024

Jay Alammar & Maarten Grootendorst. O'Reilly. Used: Ch. 6 "Prompt Engineering": "Output Verification".

How is ChatGPT's behavior changing over time? · Lingjiao Chen et al., 2023

Lingjiao Chen, Matei Zaharia & James Zou. arXiv:2307.09009. Used: Abstract (fetched 2026-10-21). Open source