Mid-level interview question
Two annotators grading the same chatbot replies keep disagreeing. What do you change?
Model answer
The instructions, not the annotators. Turn each criterion into a rubric with examples: what a reply that earns each score looks like, and why. Then have two people grade the same sample: their disagreements show where the rubric is ambiguous. Agreement alone isn't enough, since both can pass a reply that invents a price unless the rubric checks facts.
Practise the rest
10 Success criteria questions, 492 in all, each with a model answer. Free.
More Success criteria interview questions
- What is evaluation-driven development?
- What kinds of criteria would you use to evaluate an LLM application?
- Your support bot's factual consistency is 90%. Is that good enough to automate support?
- Version B has a higher pass rate than version A on last week's production logs. Do you ship B?
- How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?
- The prompt mixes a regression target with precision and recall: untangle it, and pick metrics and a modeling plan tied to the real decision.