Design a system that validates the quality of an LLM's outputs.

Two loops. Offline: a versioned eval set scored by cheap deterministic checks first, like schema, required facts and banned content, then an LLM judge calibrated against human labels, gating every change. Online: the same checks on live outputs, sampled judge scores, and user signals such as retries, sliced by task. Failures flow back into the eval set, and humans review a sample so the judge doesn't drift.