Topic 5

Where structure comes from

AI Engineering (Ch. 2, "Structured OutputsStructured outputModel output in a fixed machine-readable format, such as JSON with known keys or one label from a list, so code can parse and use it.Where this is taught") lists where in the stack you can push a model towards a format: prompting, post-processing, test time compute, constrained samplingConstrained samplingAt each generation step, masking every token that would break a rule, grammar or schema, then renormalising the remaining probabilities before sampling. Guarantees format, not correctness.Where this is taught and finetuning. The first three, it says, "are more like bandages"; "for intensive treatment, you need constrained sampling and finetuning."

• Prompting: instructions and examples (the earlier pages). No guarantee, and "a few percentage points of invalid model outputs can still be unacceptable for many applications."
• Post-processing: a script fixes the mistakes the model keeps making. LinkedIn's defensive YAML parser took correct YAML outputs from 90% to 99.99% (Bottaro and Ramgopal, 2020, as cited). It only works when mistakes are small and easy to fix.
• Test time compute: keep generating until an output fits.
• Constrained samplingConstrained samplingAt each generation step, masking every token that would break a rule, grammar or schema, then renormalising the remaining probabilities before sampling. Guarantees format, not correctness.Where this is taught: at each step, remove the tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught the format doesn't allow, then sample from the rest.
• Finetuning on examples in your format: "the most effective and general approach", much more reliable than prompting but still no guarantee. A classifier head can guarantee one of the classes.

The scores are GPT-2's real logitsLogitA raw score the model gives each vocabulary token for being next. Softmax turns logits into probabilities.Where this is taught from Module 01's sampling lesson; we applied the filter to those recorded scores rather than rerunning the model. For JSON the filter is a grammar ("after { you can't have another { unless it's part of a string"). The book's costs: every format needs its own grammar, building one is nontrivial, and checking it can add latency.