Mid-level interview question
You add a repair step that keeps the first JSON object and closes open braces. Nearly every output now parses, but half are still invalid. Why?
Model answer
Repair fixes syntax, never content. On the lab's ten hand-written, illustrative outputs, repair lifts parsing from 50% to 90%, but still only 50% are valid: the wrong type, invented label, missing field and long summary were valid JSON all along, and a cut-off summary can't be finished.
Scroll sideways to see the whole diagram
The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
Text version
- The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
- If parsing fails, a cheap repair may save it: strip the preamble or the code fence, add a missing brace, then check again.
- If the content is wrong (a missing field, an invented label), repair can't help. Call the model again, ideally telling it what failed.
- Cap the retries. After the last one, fall back, flag the input for a person and log the bad output: it is your best data on how outputs break.
After AI Engineering, Ch. 2, "Structured Outputs" (post-processing; test time compute: generate until an output fits). Illustrative flow.
Content failures need a retry with the validator's error, capped, then a logged fallback.
Practise the rest
10 Structured outputs questions, 492 in all, each with a model answer. Free.
More Structured outputs interview questions
- What is in-context learning, and what does each example in the prompt cost you?
- Your API's JSON mode is on. In what ways can an output still break the code that reads it?
- Your few-shot sentiment prompt lists six positive examples, then two negative ones at the very end. What's the risk, and what would you do?
- Each attempt gives valid output 90% of the time. A colleague proposes up to three attempts to reach 99.9%. What do you say?
- Where in the stack can you push a model towards a format, and what would you use for a classifier that must never invent a label?
- Zero-shot or few-shot prompting: what's the difference, and when do you add examples?