Mid-level interview question
Explain one step of constrained sampling. What does it guarantee, and what doesn't it?
Model answer
Score every token, mask the ones the rule forbids, renormalise the survivors, then sample as usual. It guarantees form, not truth.
Scroll sideways to see the whole diagram
The model scores every token, as in Module 01. Here GPT-2 continues "The largest planet in our solar system is".
Text version
- The model scores every token, as in Module 01. Here GPT-2 continues "The largest planet in our solar system is".
- Before sampling, every token that would break the rule is removed: its probability becomes exactly 0.
- The allowed tokens are rescaled to sum to 1. Their order doesn't change: " Earth" stays ahead of " Jupiter".
- Sampling then works as before, but can only pick an allowed token. Format is guaranteed; content is not.
After AI Engineering, Ch. 2 "Structured Outputs" > "Constrained sampling". Probabilities: GPT-2 small, lib/labs/data/sampling.json.
On "The largest planet in our solar system is", GPT-2's recorded favourite is " about" (9.4%). Allowing only planet names, " Earth" (2.8% before masking) edges out " Jupiter" (2.7%); renormalised, 24.8% to 23.7%, upper bounds since only the top 50 tokens are named. Renormalising keeps the survivors' order.
Practise the rest
8 Tools & constraints questions, 492 in all, each with a model answer. Free.
More Tools & constraints interview questions
- In tool calling, who executes the tool, and what does the model need in order to use the result?
- The model keeps calling search_orders for refund questions that another tool handles. What do you change first?
- Your agent has a delete_records tool. What has to happen between the model emitting a call and the delete running?
- Should you turn on grammar-constrained decoding for every structured output? What would you watch?
- Implement a client that spreads LLM calls across several providers, respecting their rate limits and failing over when one is down.
- Design an AI chat for a spreadsheet product that turns commands like sum column K or delete row 5 into validated tool calls on the sheet.