Topic 1

Exact scores and subjective ones

Once you know what good means, you need a way to score outputs against it. AI Engineering (Ch. 3, "Exact Evaluation") starts with one distinction: exact evaluation gives a verdict with no ambiguity, and subjective evaluation depends on who, or what, is judging.

A multiple-choice answer is exact: if the answer is A and you picked B, you're wrong. An essay grade is subjective: it depends on the grader, and the same grader, asked twice some time apart, can score the same essay differently. Clear guidelines make subjective grading more consistent, but never fully exact.

Predict first

Which of these gives an exact score?

A teammate rating a summary from 1 to 5Running generated code against its unit testsA model judging which of two replies is more helpfulUsers clicking thumbs up or down

Three ways to score an outputStep 1 / 6

Scroll sideways to see the whole diagram

You have an output to score.

Text version

You have an output to score.
Best case: run it and check it does the job. Code against its tests, a game bot by its score.
If you can't run it, compare it to a known good answer: word for word, by the words it shares, or by meaning.
Otherwise, ask a judge. That's the next lesson.
The first two are exact: the same output always gets the same score. A judge's score can change with the judge and its prompt.
Use the most exact method your task allows.

This lesson covers the exact approaches the book describes: checking that an output does its job, and comparing it with known good answers. Models as judges are subjective, and they get the next lesson.

Quick check

Why prefer an exact method when your task allows one?

The same output always gets the same score, so a change in score means the output changedIt's always more accurateIt needs no test dataIt's the only kind that scales