# Observability and incidents interview questions > 6 observability and incidents interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/observability Last updated: 2026-10-23 6 questions: 1 junior, 3 mid-level and 2 senior. Taught in [Observability and Incident Analysis](https://llmdojo.dev/learn/production-ai-systems/observability-and-incidents). ## [What are MTTD, MTTR and change failure rate?](https://llmdojo.dev/interview/q/what-are-mttd-mttr-and-change-failure-rate) Level: Junior **Mean time to detection**: how long until you notice. **Mean time to response**: how long to resolve. **Change failure rate**: the share of changes needing a fix or rollback. Together they measure how observable a system is. ## [How do you design production metrics for an AI feature?](https://llmdojo.dev/interview/q/how-do-you-design-production-metrics-for-an-ai-feature) Level: Mid-level Start from the **failures** you want to catch, not what's easy to plot: format failures, factual consistency, safety and refusals, user signals, latency and cost. Make every metric breakable by user, release, prompt version and time. **Diagram: What to measure in production** 1. AI Engineering: format failures are the easiest to track: how often outputs are invalid, and how many could be fixed. 2. Quality and safety: factual consistency and relevance, often with AI judges; toxicity, private data, how often guardrails fire and how often the system refuses. 3. User behaviour: how often generations are stopped, turns per conversation, input and output length over time. 4. Latency and cost, as in lesson 3. 5. The book's two rules: metrics aren't the goal, so design them around the failure modes you care about; and make them breakable by users, releases, prompt versions and time. ## [Quality dropped after a canary rollout. How do you find what broke?](https://llmdojo.dev/interview/q/quality-dropped-after-a-canary-rollout-how-do-you-find-what-broke) Level: Mid-level Break the metric down: by release on the same days, then by query type. In the course's synthetic incident, only how-to questions on the new release got worse, 3% to 12% thumbs-down. Then read **traces** of affected requests to find the failing step: there, retrieval returned a chunk cut off mid-procedure. ## [What implicit feedback signals would you track in a chat product?](https://llmdojo.dev/interview/q/what-implicit-feedback-signals-would-you-track-in-a-chat-product) Level: Mid-level Early termination; error corrections like “No, …” or “I meant, …”; rephrasing; requests like “Are you sure?” or “Show me the sources”; and users editing outputs. They're noisier than thumbs but far more plentiful. And they're user data, with privacy obligations. ## [Metrics are stable, yet you suspect drift. What drifts in an AI system?](https://llmdojo.dev/interview/q/metrics-are-stable-yet-you-suspect-drift-what-drifts-in-an-ai) Level: Senior - **Prompts**: a template changes, someone fixes a typo. - **Users**: phrasing and topics shift; the book's example is shrinking response lengths as users ask for concise answers. - **Models**: an API's model is updated. **Diagram: Three things that drift** 1. AI Engineering: a system prompt can change without you knowing: a template it's built on was updated, or a colleague fixed a typo. Simple logic can catch that. 2. Users adapt. The book's example: people learn to ask for concise answers, so response length drops gradually. Metrics alone won't say why; it needs investigation. 3. And the model behind an API can change, as lesson 1 showed. 4. Version prompts and models so changes are visible, watch trends over weeks as well as spikes, and rerun the evaluation set on a schedule. Version everything, watch slow trends and new input slices, and rerun the evaluation set on a schedule. ## [Walk me through handling an AI incident.](https://llmdojo.dev/interview/q/walk-me-through-handling-an-ai-incident) Level: Senior Detect; **mitigate first** (roll back, flag off, fall back); then diagnose with breakdowns and traces; fix and add the failures to the test set; write a **blameless postmortem** covering impact, timeline, causes and follow-ups, asking why the system allowed it, not who erred. ## Related - [All interview questions](https://llmdojo.dev/interview)