Topic 4

Rubrics with examples

A criterion becomes usable when it has a rubric: a scoring system and an example of what earns each score. AI Engineering (Ch. 4, "Step 2. Create an Evaluation Guideline", "Create scoring rubrics with examples") describes the choices:

Binary: 0 or 1. Some teams score factual consistency as consistent or not.
Three values: −1 when the output contradicts the context, 0 when it's neutral, 1 when the context supports it.
A scale, such as 1 to 5, or a number between 0 and 1.

Which to use depends on your data and what you'll do with the score. Whichever you choose, write down what a reply that earns each score looks like, and why.

Predict first

Graders are told only “Rate each reply as good or bad.” What's the single most useful thing to add?

A finer scale, 1 to 10The criteria, each with an example of a reply that passes and one that failsMore gradersA time limit per reply

Two people grade the same ten replies

Illustrative replies and grades · not from a real model

Same grade

5 of 10

Agreement

50.0%

How do I export my data?
“Go to Settings → Data → Export. You'll get an email with a download link.”
A good B good
Why was I charged twice?
“Charges can happen for many reasons. Is there anything else I can help with?”
A good B bad
Can I use the app offline?
“Yes. Offline mode works on mobile, and changes sync when you reconnect.”
A good B good
Reset my password, please.
“Done! I've reset your password and emailed you the new one.”
A good B bad
Who's going to win the election?
“I can only help with questions about the app.”
A bad B good
Is there a student discount?
“Yes, 50% off with a school email.”
A good B good
How do I delete my account?
“Settings → Account → Delete account. This can't be undone.”
A good B good
The app crashes when it starts.
“Have you tried restarting your phone?”
A bad B good
Do you support single sign-on?
“Yes, on the Business plan, through SAML.”
A good B good
Cancel my subscription.
“Ugh, fine. Settings → Billing → Cancel.”
A good B bad

Agreement is the share of replies both people graded the same way. High agreement is necessary but not enough: with only “Is this good?”, both called the invented 50% discount good.

Two people grading the same ten replies agree on 5 when all they're told is "Is this a good reply?". With the three-line rubric they agree on 9. The one left is a genuine judgement call: is "Have you tried restarting your phone?" an answer to "The app crashes"? A disagreement like that is useful: it shows exactly where the rubric needs one more sentence.

The book's advice is to validate the rubric with people, such as yourself, coworkers or friends. If they find it hard to follow, refine it until it's unambiguous. That can take several rounds. The same guideline can later be reused to label training data.

Quick check

Under the vague instruction, both graders called the “50% student discount” reply good, though the docs say 20%. What does that show?

The reply was goodThe graders were carelessBinary scores are too coarseAgreement isn't the same as being right: the rubric has to include the facts that make a reply wrong