Observability and incidents interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 observability and incidents questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Observability and Incident Analysis teaches the ideas behind them.Updated
Model answer
Mean time to detection: how long until you notice. Mean time to response: how long to resolve. Change failure rate: the share of changes needing a fix or rollback. Together they measure how observable a system is.
Model answer
Start from the failures you want to catch, not what's easy to plot: format failures, factual consistency, safety and refusals, user signals, latency and cost. Make every metric breakable by user, release, prompt version and time.
Scroll sideways to see the whole diagram
AI Engineering: format failures are the easiest to track: how often outputs are invalid, and how many could be fixed.
Text version
- AI Engineering: format failures are the easiest to track: how often outputs are invalid, and how many could be fixed.
- Quality and safety: factual consistency and relevance, often with AI judges; toxicity, private data, how often guardrails fire and how often the system refuses.
- User behaviour: how often generations are stopped, turns per conversation, input and output length over time.
- Latency and cost, as in lesson 3.
- The book's two rules: metrics aren't the goal, so design them around the failure modes you care about; and make them breakable by users, releases, prompt versions and time.
Model answer
Break the metric down: by release on the same days, then by query type. In the course's synthetic incident, only how-to questions on the new release got worse, 3% to 12% thumbs-down. Then read traces of affected requests to find the failing step: there, retrieval returned a chunk cut off mid-procedure.
Model answer
Early termination; error corrections like “No, …” or “I meant, …”; rephrasing; requests like “Are you sure?” or “Show me the sources”; and users editing outputs. They're noisier than thumbs but far more plentiful. And they're user data, with privacy obligations.
Model answer
- Prompts: a template changes, someone fixes a typo.
- Users: phrasing and topics shift; the book's example is shrinking response lengths as users ask for concise answers.
- Models: an API's model is updated.
Scroll sideways to see the whole diagram
AI Engineering: a system prompt can change without you knowing: a template it's built on was updated, or a colleague fixed a typo. Simple logic can catch that.
Text version
- AI Engineering: a system prompt can change without you knowing: a template it's built on was updated, or a colleague fixed a typo. Simple logic can catch that.
- Users adapt. The book's example: people learn to ask for concise answers, so response length drops gradually. Metrics alone won't say why; it needs investigation.
- And the model behind an API can change, as lesson 1 showed.
- Version prompts and models so changes are visible, watch trends over weeks as well as spikes, and rerun the evaluation set on a schedule.
Version everything, watch slow trends and new input slices, and rerun the evaluation set on a schedule.
Model answer
Detect; mitigate first (roll back, flag off, fall back); then diagnose with breakdowns and traces; fix and add the failures to the test set; write a blameless postmortem covering impact, timeline, causes and follow-ups, asking why the system allowed it, not who erred.