Senior interview question
Should you turn on grammar-constrained decoding for every structured output? What would you watch?
Model answer
Not blindly. It needs a grammar per format, can add latency, and output cut off at the length limit is still unparsable. It can also override the model, so watch the mass on allowed tokens. On "I opened the box and found", the digits in GPT-2's top 50 held 1.5%, and the top one, " 2", was its 20th choice; " two" had 7.7 times the probability of " 2". When that mass is small, ask for the format the model writes naturally, or convert afterwards.
Practise the rest
8 Tools & constraints questions, 492 in all, each with a model answer. Free.
More Tools & constraints interview questions
- In tool calling, who executes the tool, and what does the model need in order to use the result?
- The model keeps calling search_orders for refund questions that another tool handles. What do you change first?
- Explain one step of constrained sampling. What does it guarantee, and what doesn't it?
- Your agent has a delete_records tool. What has to happen between the model emitting a call and the delete running?
- Implement a client that spreads LLM calls across several providers, respecting their rate limits and failing over when one is down.
- Design an AI chat for a spreadsheet product that turns commands like sum column K or delete row 5 into validated tool calls on the sheet.