Topic 2

Criteria drift: grading changes the rubric

Lesson 1 said to write criteria before building, and it still holds. But reading outputs keeps changing what those criteria should be. Shankar et al. (2024), studying how people grade model outputs with model-assisted evaluators, name this criteria drift: you need criteria to grade outputs, but grading outputs is how you find out what the criteria are.

Predict first

While reading replies, you notice several quote a price that's out of date, and nothing in your rubric covers that. What do you do?

Ignore it: the rubric is fixedStart a new rubric from scratchAdd a criterion for it, version the rubric, and re-score past outputs so the results stay comparableLower the pass bar

Criteria driftStep 1 / 4

Scroll sideways to see the whole diagram

You need criteria to grade outputs.

Text version

You need criteria to grade outputs.
But grading outputs is how you discover what the criteria should be. Shankar et al. call this criteria drift.
Reading real outputs shows failure types you hadn't thought to write down.
So expect the rubric to change, and version it so results stay comparable.

The paper found that some criteria seemed to depend on the specific outputs people saw. So evaluation can't be fully specified in advance and then left alone. That doesn't undo evaluation-driven development; it adds a loop to it. AI Engineering (Ch. 4, "Iterate") makes the same point: criteria, rubrics and examples change as needs and users change. Keep enough consistency, and log enough, that results still mean something across versions.

Quick check

What does criteria drift mean for your rubric?

Write it once and never change itLet each grader decideUse only numeric scoresExpect to refine it as you read outputs, and version each change