Senior interview question
Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?
Model answer
Agreement alone can't say. Break it down: of the replies people called bad, how many does the judge let through? Easy good replies inflate agreement, so a lenient judge can agree 80% and still pass most bad ones. Decide which mistake costs more, fix the rubric or examples where they disagree, and re-check whenever the judge's model, prompt or settings change.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More AI judges interview questions
- What is LLM-as-a-judge, and why is it so widely used?
- How would you write the prompt for an LLM judge?
- Your judge gives the same reply different scores on different runs. What do you do?
- A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
- What biases do LLM judges have, and how do you mitigate them?
- Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?