# Models as judges interview questions > 8 models as judges interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/ai-judges Last updated: 2026-10-23 8 questions: 1 junior, 4 mid-level and 3 senior. Taught in [Using a Model as a Judge](https://llmdojo.dev/learn/evaluation-experimentation/ai-as-a-judge). ## [What is LLM-as-a-judge, and why is it so widely used?](https://llmdojo.dev/interview/q/what-is-llm-as-a-judge-and-why-is-it-so-widely) Level: Junior Using a model to score outputs. It's fast, cheap next to human graders, works **without reference answers** so it can run on live traffic, and can judge any criterion you can describe. **Diagram: A judge is a model plus a prompt** 1. The judge gets what it needs to judge: the question, the answer, and sometimes a reference. 2. Three things decide its verdict: the model, the prompt that tells it how to judge, and its sampling settings. 3. It returns a score, and it can explain why, which helps when you audit its results. 4. Change the model, the prompt or the settings and you have a different judge, whose scores aren't comparable with the old one's. A judge is the **model plus its prompt and settings**: change any one and you have a different judge. ## [How would you write the prompt for an LLM judge?](https://llmdojo.dev/interview/q/how-would-you-write-the-prompt-for-an-llm-judge) Level: Mid-level State the **task**, the **criteria** in detail, and the **scoring system**. Prefer labels, or a short discrete scale like 1 to 5, over wide or continuous ranges, and include **an example of each score** with the reason. Ask for the reasoning too, so verdicts can be audited. ## [Your judge gives the same reply different scores on different runs. What do you do?](https://llmdojo.dev/interview/q/your-judge-gives-the-same-reply-different-scores-on-different-runs-what) Level: Mid-level Reduce the randomness: fix the **sampling settings**, such as temperature 0, and add **scored examples** to the prompt. Zheng et al. raised GPT-4's consistency from 65% to 77.5% that way. But consistent isn't **accurate**, and examples make every call longer and costlier, so check the judge against human labels too. ## [A vendor reports “faithfulness 4.2” and your pipeline reports “faithfulness 0.8”. Which system is more faithful?](https://llmdojo.dev/interview/q/a-vendor-reports-faithfulness-4-2-and-your-pipeline-reports-faithfulness-0) Level: Mid-level You can't tell. Judge criteria **aren't standardised**: tools ship a criterion called faithfulness with different prompts and scales, such as 1 to 5, 0 or 1, or yes or no. A score is defined by its **judge model, prompt and settings**, so compare only scores from the same judge, and re-score a baseline whenever the judge changes. ## [What biases do LLM judges have, and how do you mitigate them?](https://llmdojo.dev/interview/q/what-biases-do-llm-judges-have-and-how-do-you-mitigate-them) Level: Senior **Self-bias**: favouring its own outputs, so don't let a model judge itself against a competitor. **First-position bias**: favouring the first answer, so judge each pair **in both orders** and call disagreements ties. **Verbosity bias**: preferring longer answers, even ones with errors, so compare verdicts with length and ask for brevity in the rubric. Different judges have different biases: measure yours. ## [Your judge agrees with human graders 80% of the time. Can you use it as a quality gate?](https://llmdojo.dev/interview/q/your-judge-agrees-with-human-graders-80-of-the-time-can-you) Level: Senior Agreement alone can't say. Break it down: of the replies people called **bad**, how many does the judge **let through**? Easy good replies inflate agreement, so a lenient judge can agree 80% and still pass most bad ones. Decide which mistake costs more, fix the rubric or examples where they disagree, and re-check whenever the judge's model, prompt or settings change. ## [Design a hybrid evaluation platform that combines human reviewers with LLM judges. How do you calibrate the judges and measure agreement between evaluators?](https://llmdojo.dev/interview/q/design-a-hybrid-evaluation-platform-that-combines-human-reviewers-with-llm-judges) Level: Senior Layer it. **Deterministic checks** run on everything, LLM judges score the rest, and humans label a stratified sample plus cases where judges are unsure or disagree. Humans are the reference. Each judge, meaning model, prompt and settings, is **calibrated against a gold set** and promoted only when its agreement with humans, by Cohen's kappa, nears the agreement between humans. Low inter-annotator agreement means the rubric is unclear. Re-validate judges on fresh labels regularly and whenever the judge or product changes, and report per slice with confidence intervals. ## [Design an evaluation platform for a design-editing agent that handles both exact property changes and open-ended visual requests, combining hard assertions, rubrics and calibrated evaluators.](https://llmdojo.dev/interview/q/design-an-evaluation-platform-for-a-design-editing-agent-that-handles-both) Level: Mid-level Split tasks by verifiability. Exact requests, like setting a padding, get **hard assertions** on the document diff: the target property changed, and nothing else did. Open-ended requests get a multimodal judge over before and after renders, scored per **rubric criterion**: intent followed, content preserved, layout intact. **Calibrate** each judge against designer labels and track agreement over time. The platform: versioned files and instructions, sandboxed runs, stored traces, per-slice results, and a regression gate. ## Related - [All interview questions](https://llmdojo.dev/interview)