Senior interview question
What biases do LLM judges have, and how do you mitigate them?
Model answer
Self-bias: favouring its own outputs, so don't let a model judge itself against a competitor. First-position bias: favouring the first answer, so judge each pair in both orders and call disagreements ties. Verbosity bias: preferring longer answers, even ones with errors, so compare verdicts with length and ask for brevity in the rubric. Different judges have different biases: measure yours.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More AI judges interview questions
- What is LLM-as-a-judge, and why is it so widely used?
- How would you write the prompt for an LLM judge?
- Your judge gives the same reply different scores on different runs. What do you do?
- A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
- Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?
- Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?