Mid-level interview question
Your support bot's factual consistency is 90%. Is that good enough to automate support?
Model answer
It depends on the slices and the bar. Map the score to a business outcome: what share of requests can be automated at this level? Then check each slice: an overall 90% can hide a much lower score on billing questions, below your usefulness threshold there, while product questions are well above it.
Practise the rest
10 Success criteria questions, 492 in all, each with a model answer. Free.
More Success criteria interview questions
- What is evaluation-driven development?
- What kinds of criteria would you use to evaluate an LLM application?
- Two annotators grading the same chatbot replies keep disagreeing. What do you change?
- Version B has a higher pass rate than version A on last week's production logs. Do you ship B?
- How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?
- The prompt mixes a regression target with precision and recall: untangle it, and pick metrics and a modeling plan tied to the real decision.