Mid-level interview question
Your judge gives the same reply different scores on different runs. What do you do?
Model answer
Reduce the randomness: fix the sampling settings, such as temperature 0, and add scored examples to the prompt. Zheng et al. raised GPT-4's consistency from 65% to 77.5% that way. But consistent isn't accurate, and examples make every call longer and costlier, so check the judge against human labels too.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More AI judges interview questions
- What is LLM-as-a-judge, and why is it so widely used?
- How would you write the prompt for an LLM judge?
- A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
- What biases do LLM judges have, and how do you mitigate them?
- Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?
- Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?