Topic 1
Decide how you'll judge it, first
Before you build an AI feature, decide how you will know whether it works. It sounds obvious, and it is the step most teams skip.
AI Engineering (Ch. 4, "Evaluation Criteria") opens with a question the author put to conference audiences: what's worse, an application that was never deployed, or one that's deployed but that nobody can tell is working?
Predict first
The book calls the fix evaluation-driven development, after test-driven development in software, where you write the tests before the code. Here it means writing down the evaluation criteria before building.
Scroll sideways to see the whole diagram
Start from the job, not the model: what is this application for?
Text version
- Start from the job, not the model: what is this application for?
- Write down what a good output must be, and what it must never do.
- Turn each criterion into a score with examples, so two people would grade the same output the same way.
- Then choose the test sets and the methods that apply the rubric.
- Building comes last. Every change is then judged by a plan you made before you saw the results.
It also explains which AI applications are common in production: the ones with clear ways to measure success. Recommenders have engagement and purchases, fraud detection has money saved, code has tests that pass or fail, and classification has right and wrong labels. Applications that are hard to evaluate are harder to justify, and some useful ones never get built for that reason.
This module is about making evaluation possible for the open-ended cases too. This lesson starts at the top of that plan: deciding what "good" means for your application.
Quick check