You will be able to
- Read real outputs, failing and passing, before choosing a fix, and inspect them by source, slice and time.
- Explain criteria drift and why a rubric is versioned and refined as outputs are graded.
- Write free-form notes on failures, group them into failure modes, and count each mode.
- Prioritise failure modes by frequency, severity and cost to fix.
- Trace a failure to the step that caused it (retrieval, generation or source) using logs.
- Turn each failure into a test case in a known-failures set kept apart from production-like data.
- Design production metrics around the failure modes you care about.
Topics
Your review
Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.
Sign in freeSources
AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 4 "Evaluate AI Systems": "Design Your Evaluation Pipeline" > "Step 1. Evaluate All Components in a System", "Step 3" > "Annotate evaluation data"; "Iterate". Ch. 6 "RAG and Agents": "Agent Failure Modes and Evaluation". Ch. 8 "Dataset Engineering": "Data Processing" > "Inspect Data". Ch. 10 "AI Engineering Architecture and User Feedback": "Monitoring and Observability".
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences · Shreya Shankar et al., 2024
Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran & Ian Arawjo. arXiv:2404.12272. Used: Abstract (fetched 2026-10-22): "criteria drift". Open source