Topic 5

Find the step that failed

Knowing a failure's category isn't the same as knowing its cause. Lesson 1 said to score each component; error analysis is where that pays off. For an assistant that answers from documents, three questions usually find the step that failed:

Which step failed?Step 1 / 5

Scroll sideways to see the whole diagram

An answer was wrong. Before fixing anything, find out where.

Text version

An answer was wrong. Before fixing anything, find out where.
If the right passage never reached the model, no prompt change will help: fix retrieval.
If it was there but the answer ignored or contradicted it, fix the prompt or the model.
If the answer followed a passage that was itself wrong or out of date, fix the source.
All three questions need the retrieved passages and the prompt logged with every answer.

Predict first

The assistant quoted the old price of $8. Its logs show it retrieved the current pricing page, which says $10. Which step failed?

Generation: the right passage was there and the answer didn't follow itRetrievalThe docsCan't tell

This is the same diagnosis AI Engineering (Ch. 4, "Step 1. Evaluate All Components in a System") recommends: score each step on its own, so a failure points at one. In practice that needs logging. Store each answer with the passages retrieved, the prompt version and the model, as Module 03 said for every output. Without them, the three questions can't be answered.

The fix differs completely by step. Better retrieval won't help a model that ignores what it's given, and a better prompt won't help when the help page itself is wrong.

Quick check

An answer is wrong because the help page it relied on is out of date. What's the fix?

Update the help page, and add a check for stale contentA bigger modelA stricter promptBetter retrieval