Junior interview questionAsked at Scale AI · interview write-up
Design an LLM API pipeline that prompts for structured output and validates what comes back.
Model answer
Template the prompt with the schema and one example, and use the API's JSON or structured-output mode if it has one. Parse the reply and validate it with Pydantic against the same schema. On failure, retry with the validation error, a few times at most, then fail loudly.
Scroll sideways to see the whole diagram
The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
Text version
- The happy path: the text parses, every field passes its check, and the value goes on to the code that needs it.
- If parsing fails, a cheap repair may save it: strip the preamble or the code fence, add a missing brace, then check again.
- If the content is wrong (a missing field, an invented label), repair can't help. Call the model again, ideally telling it what failed.
- Cap the retries. After the last one, fall back, flag the input for a person and log the bad output: it is your best data on how outputs break.
After AI Engineering, Ch. 2, "Structured Outputs" (post-processing; test time compute: generate until an output fits). Illustrative flow.
Practise the rest
10 Structured outputs questions, 492 in all, each with a model answer. Free.
More from Scale AI interview questions.
More Structured outputs interview questions
- What is in-context learning, and what does each example in the prompt cost you?
- Your API's JSON mode is on. In what ways can an output still break the code that reads it?
- Your few-shot sentiment prompt lists six positive examples, then two negative ones at the very end. What's the risk, and what would you do?
- You add a repair step that keeps the first JSON object and closes open braces. Nearly every output now parses, but half are still invalid. Why?
- Each attempt gives valid output 90% of the time. A colleague proposes up to three attempts to reach 99.9%. What do you say?
- Where in the stack can you push a model towards a format, and what would you use for a classifier that must never invent a label?