Mid-level interview question
How do you design production metrics for an AI feature?
Model answer
Start from the failures you want to catch, not what's easy to plot: format failures, factual consistency, safety and refusals, user signals, latency and cost. Make every metric breakable by user, release, prompt version and time.
Scroll sideways to see the whole diagram
AI Engineering: format failures are the easiest to track: how often outputs are invalid, and how many could be fixed.
Text version
- AI Engineering: format failures are the easiest to track: how often outputs are invalid, and how many could be fixed.
- Quality and safety: factual consistency and relevance, often with AI judges; toxicity, private data, how often guardrails fire and how often the system refuses.
- User behaviour: how often generations are stopped, turns per conversation, input and output length over time.
- Latency and cost, as in lesson 3.
- The book's two rules: metrics aren't the goal, so design them around the failure modes you care about; and make them breakable by users, releases, prompt versions and time.
Practise the rest
6 Observability questions, 492 in all, each with a model answer. Free.