Mid-level interview question
Your few-shot sentiment prompt lists six positive examples, then two negative ones at the very end. What's the risk, and what would you do?
Model answer
The model may lean towards the label it saw last. Zhao et al. traced unstable few-shot accuracy to biases, including towards answers near the end of the prompt; Lu et al. found order alone can swing between near state-of-the-art and random-guess performance. Both studied GPT-3-era models, so test yours: cover every label and the hard cases, shuffle, compare a few orders on a fixed test set, and end on a marker.
Practise the rest
10 Structured outputs questions, 492 in all, each with a model answer. Free.
More Structured outputs interview questions
- What is in-context learning, and what does each example in the prompt cost you?
- Your API's JSON mode is on. In what ways can an output still break the code that reads it?
- You add a repair step that keeps the first JSON object and closes open braces. Nearly every output now parses, but half are still invalid. Why?
- Each attempt gives valid output 90% of the time. A colleague proposes up to three attempts to reach 99.9%. What do you say?
- Where in the stack can you push a model towards a format, and what would you use for a classifier that must never invent a label?
- Zero-shot or few-shot prompting: what's the difference, and when do you add examples?