Topic 7

Measuring the parse rate, and wrap-up

"It usually returns JSON" is not a number. Run the prompt over a fixed set of inputs and count three rates, each harder to pass than the last:

1. Parse rate: the share of outputs your parser accepts.
2. Valid rate: the share that also pass the schema.
3. Correct rate: the share whose values are right, checked against labelled answers.

The lab's illustrative set shows why one isn't enough: 50% parse, 10% valid. With repair, 90% parse and still only 50% valid. Report each rate with and without repair, so a parser fix can't hide a prompt problem. And keep the failing outputs: they show which fix to try.

Retries raise the valid rate, at a price.

This is a best case. Retries are independent only if failures are random, but the book notes a model "tends to repeat similar mistakes". Measure how often a retry actually succeeds on your own failing inputs.

What to keep

• Few-shot examples teach in context; stronger models need fewer, but domain-specific tasks still benefit. Each shot costs tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught on every call.
• Pick examples that cover the labels, keep them compact, end on a marker, and test the order.
• Specify keys, types and allowed labels, then validate anyway.
• Parsing catches syntax; only a schema check catches wrong types and invented labels.
• Repair fixes syntax; content failures need a retry. Constrained samplingConstrained samplingAt each generation step, masking every token that would break a rule, grammar or schema, then renormalising the remaining probabilities before sampling. Guarantees format, not correctness.Where this is taught and finetuning make the format reliable, not the values right.
• Measure parse, valid and correct rates on a fixed set.