Topic 7
Watch for failures in production
Error analysis doesn't stop at launch. AI Engineering (Ch. 10, "Monitoring and Observability") argues that metrics aren't the goal in themselves: a metric exists to tell you when something is wrong and where to improve. So before choosing what to monitor, list the failure modes you want to catch, and design metrics around them.
Predict first
Scroll sideways to see the whole diagram
Start from a failure you don't want: making things up. Track whether answers are supported by their context.
Text version
- Start from a failure you don't want: making things up. Track whether answers are supported by their context.
- Burning through API credit: track tokens per request and the cache hit rate.
- Your own error analysis adds more: here, how often no relevant passage was found.
- Metrics aren't the goal. Each one should tell you when a failure you care about is happening.
The book's other example is cost: to avoid burning through API credit, track input and output tokens per request, and the cache's cost and hit rate. Your own error analysis adds metrics no generic dashboard has. Here, the share of answers where no relevant passage was retrieved: the failure mode at the top of this lesson's count.
Production also brings signals you don't get in testing: users retrying, rephrasing or rating answers. Sample from them in each round of error analysis. And the earlier lessons still apply in production: slice the metrics, check judges against people, and keep their settings fixed.
Quick check