Observability and Incident Analysis

You will be able to

• Use metrics, logs and traces in sequence, and judge observability by MTTD, MTTR and change failure rate.
• Design production metrics around the failures you want to catch: format, quality, safety, user behaviour, latency and cost.
• Break a metric down by release, query type and other dimensions to find where a problem lives.
• Go from a metric to the traces of affected requests and find the failing pipeline step.
• Detect drift from prompt changes, user behaviour changes and model changes.
• Read explicit and implicit user feedback, and treat it as user data.
• Run an incident: mitigate first, then diagnose, fix, add tests, and write a blameless postmortem.

Topics

1. 1Seeing what production does
2. 2Metrics designed from failures
3. 3Break it down
4. 4From a metric to a trace
5. 5Drift
6. 6What users tell you
7. 7Incidents and postmortems, and wrap-up