Topic 2
Four kinds of criteria
An application's evaluation starts with a list of criteria specific to it. AI Engineering (Ch. 4, "Evaluation Criteria") sorts them into four kinds, using the task of summarising a legal contract:
- Domain capability: does the model understand the subject (here, contracts)? A model can't understand what it never saw in training, so this depends on its training data and size.
- Generation capability: is the summary coherent, and faithful to the contract? The book counts factual consistency and safety here.
- Instruction following: does the output follow the requested format and length?
- Cost and latency: what each summary costs, and how long you wait for it. Module 02 measures these in Cost per request.
Predict first
Scroll sideways to see the whole diagram
Take one task: summarising a legal contract.
Text version
- Take one task: summarising a legal contract.
- Domain capability: does the model understand the subject at all? Training data and size limit it.
- Generation: is the text coherent, and does it stay faithful to the source?
- Instruction following: the requested format and length. Cost and latency: what each output costs and how long it takes.
- These are separate checks: a summary can be accurate but too long, or perfectly formatted but wrong.
This lesson's running example is a support assistant for a software product. It answers customers' questions from the product's help docs. For it, domain capability means knowing the product. Generation means saying only what the docs support, politely. Instruction following might mean a reply short enough to read on a phone, and cost and latency still apply to every reply.
Quick check