Lesson 4 · 8 topics · 26 min · intermediate

Comparing Systems You Can Trust

You will be able to

Choose baselines for a result: random, the simplest approach, the current system, and people.
Run an ablation study one part at a time, and remove parts that don't change the score measurably.
Bootstrap an evaluation set, resampling the same cases for both versions, and report a difference with its interval.
Size a test set from the smallest difference that would change a decision, using the 3×-smaller, 10×-more rule of thumb.
Explain how pairwise matches become rankings, why Elo depends on match order, and when preference shouldn't replace correctness.
Read public leaderboards for what they can and can't show: scale, transitivity, crowd quality, and relative versus absolute performance.
Explain data contamination, how it happens, and how n-gram overlap and perplexity detect it.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology": "Ranking Models with Comparative Evaluation", "Challenges of Comparative Evaluation" ("Scalability bottlenecks", "Lack of standardization and quality control", "From comparative performance to absolute performance"). Ch. 4 "Evaluate AI Systems": "Evaluation Criteria" > "Domain-Specific Capability"; "Navigate Public Benchmarks" > "Data contamination with public benchmarks", "Handling data contamination"; "Design Your Evaluation Pipeline" > "Step 3" > "Annotate evaluation data". Ch. 6 "RAG and Agents": "Tool selection"; "Agent Failure Modes and Evaluation" > "Efficiency".

You will be able to

Choose baselines for a result: random, the simplest approach, the current system, and people.

Run an ablation study one part at a time, and remove parts that don't change the score measurably.

Bootstrap an evaluation set, resampling the same cases for both versions, and report a difference with its interval.

Size a test set from the smallest difference that would change a decision, using the 3×-smaller, 10×-more rule of thumb.

Explain how pairwise matches become rankings, why Elo depends on match order, and when preference shouldn't replace correctness.

Read public leaderboards for what they can and can't show: scale, transitivity, crowd quality, and relative versus absolute performance.

Explain data contamination, how it happens, and how n-gram overlap and perplexity detect it.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 3 "Evaluation Methodology": "Ranking Models with Comparative Evaluation", "Challenges of Comparative Evaluation" ("Scalability bottlenecks", "Lack of standardization and quality control", "From comparative performance to absolute performance"). Ch. 4 "Evaluate AI Systems": "Evaluation Criteria" > "Domain-Specific Capability"; "Navigate Public Benchmarks" > "Data contamination with public benchmarks", "Handling data contamination"; "Design Your Evaluation Pipeline" > "Step 3" > "Annotate evaluation data". Ch. 6 "RAG and Agents": "Tool selection"; "Agent Failure Modes and Evaluation" > "Efficiency".