Using a Model as a Judge

You will be able to

• Explain why AI judges are widely used, and treat a judge as a system of model, prompt and sampling settings.
• Choose between judging an answer alone, against a reference, or pairwise, for a given purpose.
• Write a judge prompt with the task, detailed criteria, a short labelled scale, and examples with reasons.
• Recognise run-to-run inconsistency and know what reduces it, and why consistent doesn't mean accurate.
• Explain why judge criteria with the same name aren't comparable across tools or prompt versions.
• Describe self, first-position and verbosity bias, and mitigate position bias by judging both orders.
• Check a judge against human labels by counting the bad outputs it lets through, and choose a judge model while managing cost, latency and privacy.

Topics

1. 1Why use a model as a judge
2. 2Three ways to judge
3. 3Writing the judge's prompt
4. 4Same input, different scores
5. 5Same name, different metric
6. 6Self, position and length biases
7. 7Check the judge against people
8. 8Which model should judge, and wrap-up