Senior interview questionAsked at Waymo · interview write-up
Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?
Model answer
Layer it. Deterministic checks run on everything, LLM judges score the rest, and humans label a stratified sample plus cases where judges are unsure or disagree.
Humans are the reference. Each judge, meaning model, prompt and settings, is calibrated against a gold set and promoted only when its agreement with humans, by Cohen's kappa, nears the agreement between humans. Low inter-annotator agreement means the rubric is unclear. Re-validate judges on fresh labels regularly and whenever the judge or product changes, and report per slice with confidence intervals.
Practise the rest
8 AI judges questions, 492 in all, each with a model answer. Free.
More from Waymo interview questions.
More AI judges interview questions
- What is LLM-as-a-judge, and why is it so widely used?
- How would you write the prompt for an LLM judge?
- Your judge gives the same reply different scores on different runs. What do you do?
- A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?
- What biases do LLM judges have, and how do you mitigate them?
- Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?