Mid-level interview question
A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
Model answer
You can't tell. Judge criteria aren't standardised: tools ship a criterion called faithfulness with different prompts and scales, such as 1 to 5, 0 or 1, or yes or no. A score is defined by its judge model, prompt and settings, so compare only scores from the same judge, and re-score a baseline whenever the judge changes.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More AI judges interview questions
- What is LLM-as-a-judge, and why is it so widely used?
- How would you write the prompt for an LLM judge?
- Your judge gives the same reply different scores on different runs. What do you do?
- What biases do LLM judges have, and how do you mitigate them?
- Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?
- Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?