Junior interview question
What is LLM-as-a-judge, and why is it so widely used?
Model answer
Using a model to score outputs. It's fast, cheap next to human graders, works without reference answers so it can run on live traffic, and can judge any criterion you can describe.
Scroll sideways to see the whole diagram
The judge gets what it needs to judge: the question, the answer, and sometimes a reference.
Text version
- The judge gets what it needs to judge: the question, the answer, and sometimes a reference.
- Three things decide its verdict: the model, the prompt that tells it how to judge, and its sampling settings.
- It returns a score, and it can explain why, which helps when you audit its results.
- Change the model, the prompt or the settings and you have a different judge, whose scores aren't comparable with the old one's.
A judge is the model plus its prompt and settings: change any one and you have a different judge.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More AI judges interview questions
- How would you write the prompt for an LLM judge?
- Your judge gives the same reply different scores on different runs. What do you do?
- A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
- What biases do LLM judges have, and how do you mitigate them?
- Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?
- Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?