Topic 6
Validate, repair, retry
Never pass a model's output downstream unchecked. Parse it, check it against the schema, and decide what to do with each failure.
Scroll sideways to see the whole diagram
The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
Text version
- The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
- If parsing fails, a cheap repair may save it: strip the preamble or the code fence, add a missing brace, then check again.
- If the content is wrong (a missing field, an invented label), repair can't help. Call the model again, ideally telling it what failed.
- Cap the retries. After the last one, fall back, flag the input for a person and log the bad output: it is your best data on how outputs break.
After AI Engineering, Ch. 2, "Structured Outputs" (post-processing; test time compute: generate until an output fits). Illustrative flow.
Post-processing is the cheap first fix. AI Engineering (Ch. 2, "Post-processing") notes that "a model tends to repeat similar mistakes across queries", so a script can fix the common ones: its example is adding a missing closing bracket. But it "works only if the mistakes are easy to fix."
Predict first
Check outputs against a schema
Deterministic validator · illustrative outputs, not from a model
Ticket (illustrative)
I was charged twice for my March invoice. Please refund the second charge.
{
"category": one of "billing", "bug", "account", "other",
"priority": an integer 1, 2 or 3,
"summary": a string of at most 80 characters
}
No other keys.Parse rate
90% (9/10)
Valid rate
50% (5/10)
Parse rate, no repair
50%
Valid rate, with repair
50%
Repair: nothing to repair
Repair fixes syntax, never content. You could coerce "2" to 2, but each such rule is a guess about what the model meant, and a guess that is wrong passes silently. For content failures, retry instead:
- Retry with the error. Send the output and the validator's message back (
category "refund" is not one of billing, bug, account, other). The book lists generating until an output fits as test time compute. - Ask a model to check. The book notes some teams use AI to validate or correct outputs, at the price of at least two model calls per output.
- Cap it. After the last retry, fall back and log the output.
Quick check