# Success criteria and rubrics interview questions > 10 success criteria and rubrics interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/success-criteria Last updated: 2026-10-23 10 questions: 2 junior, 4 mid-level and 4 senior. Taught in [Success Criteria: Deciding What Good Means](https://llmdojo.dev/learn/evaluation-experimentation/success-criteria). ## [What is evaluation-driven development?](https://llmdojo.dev/interview/q/what-is-evaluation-driven-development) Level: Junior Writing down how you'll **evaluate** an application **before you build it**, the way test-driven development writes tests before code. You decide the criteria, and how each is scored, first. Otherwise you ship something that costs money to run and that **nobody can say is working**. ## [What kinds of criteria would you use to evaluate an LLM application?](https://llmdojo.dev/interview/q/what-kinds-of-criteria-would-you-use-to-evaluate-an-llm-application) Level: Junior Four kinds, scored separately: **domain capability** (does it understand the subject?), **generation** (coherent, faithful, safe), **instruction following** (the requested format and length), and **cost and latency**. A summary can be faithful but too long, so one overall score would hide what to fix. ## [Two annotators grading the same chatbot replies keep disagreeing. What do you change?](https://llmdojo.dev/interview/q/two-annotators-grading-the-same-chatbot-replies-keep-disagreeing-what-do-you) Level: Mid-level The instructions, not the annotators. Turn each criterion into a **rubric with examples**: what a reply that earns each score looks like, and why. Then have two people grade the same sample: their disagreements show where the rubric is **ambiguous**. Agreement alone isn't enough, since both can pass a reply that invents a price unless the rubric checks facts. ## [Your support bot's factual consistency is 90%. Is that good enough to automate support?](https://llmdojo.dev/interview/q/your-support-bot-s-factual-consistency-is-90-is-that-good-enough) Level: Mid-level It depends on the slices and the bar. Map the score to a **business outcome**: what share of requests can be automated at this level? Then check each **slice**: an overall 90% can hide a much lower score on billing questions, below your **usefulness threshold** there, while product questions are well above it. ## [Version B has a higher pass rate than version A on last week's production logs. Do you ship B?](https://llmdojo.dev/interview/q/version-b-has-a-higher-pass-rate-than-version-a-on-last) Level: Senior Not on that evidence. Each version served **different traffic**: if B got the easy self-serve tickets, its overall rate is flattered by its mix. That's **Simpson's paradox**: A can win every slice and still lose overall. Compare **slice by slice**, or better, run both on **one fixed test set** with the same mix, then check the slices you care about. ## [How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?](https://llmdojo.dev/interview/q/how-would-you-design-the-evaluation-for-a-two-step-resume-parser) Level: Senior Score the **end-to-end** answer and **each step** on its own: step 1 by similarity of the extracted text to the true text, step 2 by accuracy when it's given correct text. **Diagram: Score each step, not just the end** 1. A two-step system: pull the text out of a résumé PDF, then pull the current employer out of the text. 2. When the final answer is wrong, the end-to-end score alone can't say whether step 1 or step 2 broke. 3. Score step 1 on its own: how close is the extracted text to the real text? 4. Score step 2 on correct text: how often does it name the right employer? 5. Now a failure points at a step. The same applies to turns and whole tasks in a conversation. Then a failure points at a step, not the whole system. Add slices such as file formats and layouts, include out-of-scope inputs, and log each step's output so failures can be traced. ## [The prompt mixes a regression target with precision and recall: untangle it, and pick metrics and a modeling plan tied to the real decision.](https://llmdojo.dev/interview/q/the-prompt-mixes-a-regression-target-with-precision-and-recall-untangle-it) Level: Mid-level Ask what decision the output drives. Approve or decline is binary, so use **precision and recall at the operating threshold**, set by error costs. How much, like a loss amount, is regression, judged by MAE or RMSE. Often it's both: predict the probability of default and the loss if it happens, and combine them into **expected value per decision**. Agree the metric with stakeholders before modelling, and validate on a time-based split. ## [When would you use unsupervised methods for fraud detection, and how would you evaluate them without reliable labels?](https://llmdojo.dev/interview/q/when-would-you-use-unsupervised-methods-for-fraud-detection-and-how-would) Level: Senior Use them when labels are **scarce, delayed or blind to new attacks**: isolation forests, autoencoder reconstruction error or graph clustering flag what's unusual, not what's known fraud. Evaluate by proxy. Backtest on the labels you do have with **precision at the top k**, since investigators can review only so many. Send flagged and random cases to analysts to estimate precision and lift. Inject synthetic anomalies to check recall, and track alert volume and stability over time. Unusual isn't fraud, so feed confirmed cases into a **supervised model**. ## [You add a fixed amount to a logistic regression's intercept. What happens to the decision boundary, recall, precision, ranking metrics and calibration?](https://llmdojo.dev/interview/q/you-add-a-fixed-amount-to-a-logistic-regression-s-intercept-what) Level: Mid-level It adds the same constant to every logit, a **monotonic shift**, so the order is unchanged and **ranking metrics like AUC stay identical**. At a fixed threshold the boundary moves: a positive shift labels more examples positive, so recall rises or holds and precision usually falls. **Calibration** breaks, since predicted probabilities no longer match observed rates. It's equivalent to moving the threshold, which is how prior correction after resampling works. ## [You're shipping a classifier for 7-day purchase with 3% positives, a 10-day label delay and asymmetric business values. How do you calibrate it and pick the threshold?](https://llmdojo.dev/interview/q/you-re-shipping-a-classifier-for-7-day-purchase-with-3-positives) Level: Senior Validate **out of time**, only on examples whose labels have matured. With 3% positives, report PR-AUC, not accuracy. **Calibrate** with Platt or isotonic regression on a recent held-out window, check reliability curves and log loss, and undo any resampling first. Then act when expected value is positive: with gain G from a true positive and cost C from a false positive, the threshold is **C / (C + G)**. Better still, sweep thresholds for expected profit with confidence intervals under capacity limits. Until labels arrive, monitor score distributions. ## Related - [All interview questions](https://llmdojo.dev/interview)