Topic 3
Correct is not the same as good
Often the hardest part of evaluation isn't scoring an output; it's deciding what "good" means. AI Engineering (Ch. 4, "Design Your Evaluation Pipeline", "Step 2. Create an Evaluation Guideline", "Define evaluation criteria") recounts LinkedIn's experience with an AI job-assessment feature: the reply "You are a terrible fit" might be correct, but it isn't helpful. A good reply explains the gap between the job's requirements and the candidate's background, and what the candidate could do about it.
Predict first
Scroll sideways to see the whole diagram
A customer asks about a double charge, and the assistant replies.
Text version
- A customer asks about a double charge, and the assistant replies.
- It's polite and says nothing untrue.
- But it doesn't answer what was asked.
- Only criteria written down in advance catch this: “is anything false?” would pass it.
A guideline says what the application should do, and also what it should not. For a support assistant, should it answer a question about an upcoming election? If not, the guideline has to say which inputs are out of scope, how to recognise them, and what to reply instead.
The book gives three criteria a support application might use:
- Relevance: the reply addresses the user's question.
- Factual consistency: the reply agrees with the context it was given, here the help docs.
- Safety: the reply isn't toxic.
To find criteria like these, the book suggests trying real user queries. Generate several replies to each, by hand or with a model, and decide which are good and which are bad, and why. The reasons become the criteria.
Quick check