Models as judges interview questions
8 questions with model answers
- junior 1
- mid 4
- senior 3
8 models as judges questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 4 mid-level and 3 senior. The lesson Using a Model as a Judge teaches the ideas behind them.Updated
Model answer
Using a model to score outputs. It's fast, cheap next to human graders, works without reference answers so it can run on live traffic, and can judge any criterion you can describe.
Scroll sideways to see the whole diagram
The judge gets what it needs to judge: the question, the answer, and sometimes a reference.
Text version
- The judge gets what it needs to judge: the question, the answer, and sometimes a reference.
- Three things decide its verdict: the model, the prompt that tells it how to judge, and its sampling settings.
- It returns a score, and it can explain why, which helps when you audit its results.
- Change the model, the prompt or the settings and you have a different judge, whose scores aren't comparable with the old one's.
A judge is the model plus its prompt and settings: change any one and you have a different judge.
Model answer
State the task, the criteria in detail, and the scoring system. Prefer labels, or a short discrete scale like 1 to 5, over wide or continuous ranges, and include an example of each score with the reason. Ask for the reasoning too, so verdicts can be audited.
Model answer
Reduce the randomness: fix the sampling settings, such as temperature 0, and add scored examples to the prompt. Zheng et al. raised GPT-4's consistency from 65% to 77.5% that way. But consistent isn't accurate, and examples make every call longer and costlier, so check the judge against human labels too.
Model answer
You can't tell. Judge criteria aren't standardised: tools ship a criterion called faithfulness with different prompts and scales, such as 1 to 5, 0 or 1, or yes or no. A score is defined by its judge model, prompt and settings, so compare only scores from the same judge, and re-score a baseline whenever the judge changes.
Model answer
Self-bias: favouring its own outputs, so don't let a model judge itself against a competitor. First-position bias: favouring the first answer, so judge each pair in both orders and call disagreements ties. Verbosity bias: preferring longer answers, even ones with errors, so compare verdicts with length and ask for brevity in the rubric. Different judges have different biases: measure yours.
Model answer
Agreement alone can't say. Break it down: of the replies people called bad, how many does the judge let through? Easy good replies inflate agreement, so a lenient judge can agree 80% and still pass most bad ones. Decide which mistake costs more, fix the rubric or examples where they disagree, and re-check whenever the judge's model, prompt or settings change.
Model answer
Layer it. Deterministic checks run on everything, LLM judges score the rest, and humans label a stratified sample plus cases where judges are unsure or disagree.
Humans are the reference. Each judge, meaning model, prompt and settings, is calibrated against a gold set and promoted only when its agreement with humans, by Cohen's kappa, nears the agreement between humans. Low inter-annotator agreement means the rubric is unclear. Re-validate judges on fresh labels regularly and whenever the judge or product changes, and report per slice with confidence intervals.
Model answer
Split tasks by verifiability. Exact requests, like setting a padding, get hard assertions on the document diff: the target property changed, and nothing else did.
Open-ended requests get a multimodal judge over before and after renders, scored per rubric criterion: intent followed, content preserved, layout intact. Calibrate each judge against designer labels and track agreement over time.
The platform: versioned files and instructions, sandboxed runs, stored traces, per-slice results, and a regression gate.