Senior interview question
Each attempt gives valid output 90% of the time. A colleague proposes up to three attempts to reach 99.9%. What do you say?
Model answer
The arithmetic is right only if failures are independent: 1 − 0.1³ = 99.9% valid, for 1.11 calls on average. That's a best case. The book notes a model "tends to repeat similar mistakes", so a retry on the same input may fail the same way. I'd measure how often retries actually succeed on our failing inputs, and report parse, valid and correct rates, with and without repair, so a parser fix can't hide a prompt problem.
Practise the rest
10 Structured outputs questions, 492 in all, each with a model answer. Free.
More Structured outputs interview questions
- What is in-context learning, and what does each example in the prompt cost you?
- Your API's JSON mode is on. In what ways can an output still break the code that reads it?
- Your few-shot sentiment prompt lists six positive examples, then two negative ones at the very end. What's the risk, and what would you do?
- You add a repair step that keeps the first JSON object and closes open braces. Nearly every output now parses, but half are still invalid. Why?
- Where in the stack can you push a model towards a format, and what would you use for a classifier that must never invent a label?
- Zero-shot or few-shot prompting: what's the difference, and when do you add examples?