Constrained sampling
Also called constrained decoding, grammar-constrained sampling
At each generation step, masking every token that would break a rule, grammar or schema, then renormalising the remaining probabilities before sampling. Guarantees format, not correctness.
Interview questions about Constrained sampling
- Explain one step of constrained sampling. What does it guarantee, and what doesn't it?
Score every token, mask the ones the rule forbids, renormalise the survivors, then sample as usual. It guarantees form, not truth.
- Should you turn on grammar-constrained decoding for every structured output? What would you watch?
Not blindly. It needs a grammar per format, can add latency, and output cut off at the length limit is still unparsable.
- Where in the stack can you push a model towards a format, and what would you use for a classifier that must never invent a label?
AI Engineering lists five places: prompting, post-processing and test time compute are "more like bandages"; constrained sampling and finetuning are the…
- Your LLM writes documents that must follow a strict Word formatting schema: styles, heading levels, tables. How do you make its output conform?
Don't ask the model for Word markup. Have it emit structured output, JSON with only allowed blocks such as heading with level, paragraph, list and table, each…