Examples and structured outputs interview questions

Learning a task from examples in the prompt, with no change to the weights; each example is a shot. Nothing is learned between calls, so every shot is input tokens on every request; the context window caps how many fit. For GPT-4, in Microsoft's 2023 analysis, few-shot gave only limited gains over zero-shot.

JSON mode guarantees valid JSON, not its content. Output can still be cut off at the output-token limit, which doesn't parse. And parseable outputs can still carry wrong types ("2" for an integer), invented labels, missing or extra fields, or values past a limit. Only a schema check catches those.

The model may lean towards the label it saw last. Zhao et al. traced unstable few-shot accuracy to biases, including towards answers near the end of the prompt; Lu et al. found order alone can swing between near state-of-the-art and random-guess performance. Both studied GPT-3-era models, so test yours: cover every label and the hard cases, shuffle, compare a few orders on a fixed test set, and end on a marker.

Repair fixes syntax, never content. On the lab's ten hand-written, illustrative outputs, repair lifts parsing from 50% to 90%, but still only 50% are valid: the wrong type, invented label, missing field and long summary were valid JSON all along, and a cut-off summary can't be finished.

Content failures need a retry with the validator's error, capped, then a logged fallback.

The arithmetic is right only if failures are independent: 1 − 0.1³ = 99.9% valid, for 1.11 calls on average. That's a best case. The book notes a model "tends to repeat similar mistakes", so a retry on the same input may fail the same way. I'd measure how often retries actually succeed on our failing inputs, and report parse, valid and correct rates, with and without repair, so a parser fix can't hide a prompt problem.

AI Engineering lists five places: prompting, post-processing and test time compute are "more like bandages"; constrained sampling and finetuning are the intensive treatment. LinkedIn's defensive parser took correct YAML from 90% to 99.99%, as the book cites, but post-processing works only when mistakes are easy to fix. For fixed labels, constrain sampling to them, or use a classifier head. Neither makes the label right: GPT-2 constrained to planet names picks " Earth" over " Jupiter". Still measure the correct rate on labelled examples.

Zero-shot gives only the instruction; few-shot adds worked examples of input and output. Add them when the format or judgement is hard to describe but easy to show. Each example costs tokens on every request and the model copies its quirks, so pick varied examples and measure that they help.

Don't parse generated text; read the next-token distribution after the prompt.

import torch

def p_yes(logits, yes_ids, no_ids):
    # logits: (V,) next-token logits
    lp = logits.log_softmax(-1)
    ly = torch.logsumexp(lp[yes_ids], -1)
    ln = torch.logsumexp(lp[no_ids], -1)
    return torch.sigmoid(ly - ln)

Each label has several token spellings, so combine each group with log-sum-exp, then renormalise over the two groups: a sigmoid of the difference. Threshold it after calibrating on labelled data, since raw probabilities are often skewed, and check token ids, because a leading space changes them.

Don't ask the model for Word markup. Have it emit structured output, JSON with only allowed blocks such as heading with level, paragraph, list and table, each naming a permitted style, enforced by schema-constrained decoding.

A deterministic renderer then builds the document from the template's styles. Validate rules the schema can't express, like heading order, and retry with the error. Evaluate by diffing rendered styles against the spec.

Template the prompt with the schema and one example, and use the API's JSON or structured-output mode if it has one. Parse the reply and validate it with Pydantic against the same schema. On failure, retry with the validation error, a few times at most, then fail loudly.