Topic 8
The evaluation loop, and module wrap-up
This lesson closes the module. Each lesson answered one question about evaluating an AI application, and together they form a loop:
Scroll sideways to see the whole diagram
Decide what good means, before building.
Text version
- Decide what good means, before building.
- Score it exactly where you can, and with a checked judge where you can't.
- Compare versions fairly: against a baseline, one change at a time, with an honest interval.
- Read the failures, and fix the ones that matter most.
- What you find feeds back into the criteria and the test sets. Evaluation is a loop, not a step.
- Success criteria: what good means, written down before building, with rubrics, slices and a usefulness bar.
- Scoring against references: exact scores where possible, and why looking right isn't being right.
- Using a model as a judge: judges for everything else, with their prompts, biases and limits, checked against people.
- Comparing systems: baselines, ablations, intervals and rankings, so "better" means something.
- Error analysis: reading failures, counting them, fixing what matters, and turning each fix into a test.
Predict first
Wrap-up. From this lesson, you can now:
- read real outputs before reaching for a fix, including some that passed;
- expect criteria to drift as you grade, and version the rubric;
- write notes first, then group failures into modes and count them;
- prioritise by frequency, severity and cost of the fix;
- trace a failure to the step that caused it, using logs;
- turn every failure into a test case, and keep failure sets apart from typical traffic;
- design production metrics around the failure modes you care about.
Quick check