Senior interview question
Metrics are stable, yet you suspect drift. What drifts in an AI system?
Model answer
- Prompts: a template changes, someone fixes a typo.
- Users: phrasing and topics shift; the book's example is shrinking response lengths as users ask for concise answers.
- Models: an API's model is updated.
Scroll sideways to see the whole diagram
AI Engineering: a system prompt can change without you knowing: a template it's built on was updated, or a colleague fixed a typo. Simple logic can catch that.
Text version
- AI Engineering: a system prompt can change without you knowing: a template it's built on was updated, or a colleague fixed a typo. Simple logic can catch that.
- Users adapt. The book's example: people learn to ask for concise answers, so response length drops gradually. Metrics alone won't say why; it needs investigation.
- And the model behind an API can change, as lesson 1 showed.
- Version prompts and models so changes are visible, watch trends over weeks as well as spikes, and rerun the evaluation set on a schedule.
Version everything, watch slow trends and new input slices, and rerun the evaluation set on a schedule.
Practise the rest
6 Observability questions, 492 in all, each with a model answer. Free.