Topic 2

Does it work? Functional correctness and pass@k

The most direct question is whether the output does what it was meant to. AI Engineering (Ch. 3, "Exact Evaluation", "Functional Correctness") calls this functional correctness and calls it the ultimate metric for any application. Did the generated website meet the requirements? Did the agent actually book the restaurant?

It's often hard to automate. Code is the big exception: run the generated function on test cases and check the outputs, just as unit tests do in ordinary software and as coding sites check submitted solutions. The book lists code benchmarks that work this way, such as HumanEval and MBPP, and text-to-SQL benchmarks such as Spider, BIRD-SQL and WikiSQL. Anything with a measurable objective works similarly: a game bot by its score, an energy scheduler by the energy it saves.

Because generation is random, a model can be given several tries per problem. That gives pass@k: generate k samples per problem, count a problem as solved if any of them passes all its tests, and report the share of problems solved.

Predict first

A model is scored with pass@1, then pass@10, on the same problems. How do the scores compare?

pass@1 is higherpass@10 is at least as highThey're always equalIt depends only on the tests

pass@k: ten problems, ten samples each

Illustrative pass/fail grid · not from a real model

k: samples allowed per problem1

Problems solved

3 of 10

pass@1

30.0%

Problem 1solved

Problem 2solved

Problem 3—

Problem 4—

Problem 5—

Problem 6solved

Problem 7—

Problem 8—

Problem 9—

Problem 10—

A problem counts as solved if any of its first k samples passes all its tests. More samples can only help, so pass@k never goes down as k grows.

On these ten illustrative problems, pass@1 is 30%, pass@3 is 50%, and pass@10 is 80%. Two problems are never solved however many samples you take. So always report k alongside the score: "80%" means something very different at k = 1 and at k = 10.

Quick check

Vendor A reports 80% on a code benchmark and vendor B reports 55%. What do you check before comparing them?

Which vendor is biggerWhich one writes shorter codeNothing: higher is betterThat both used the same k, the same problems and the same tests