Success criteria and rubrics interview questions

Writing down how you'll evaluate an application before you build it, the way test-driven development writes tests before code. You decide the criteria, and how each is scored, first. Otherwise you ship something that costs money to run and that nobody can say is working.

Four kinds, scored separately: domain capability (does it understand the subject?), generation (coherent, faithful, safe), instruction following (the requested format and length), and cost and latency. A summary can be faithful but too long, so one overall score would hide what to fix.

The instructions, not the annotators. Turn each criterion into a rubric with examples: what a reply that earns each score looks like, and why. Then have two people grade the same sample: their disagreements show where the rubric is ambiguous. Agreement alone isn't enough, since both can pass a reply that invents a price unless the rubric checks facts.

It depends on the slices and the bar. Map the score to a business outcome: what share of requests can be automated at this level? Then check each slice: an overall 90% can hide a much lower score on billing questions, below your usefulness threshold there, while product questions are well above it.

Not on that evidence. Each version served different traffic: if B got the easy self-serve tickets, its overall rate is flattered by its mix. That's Simpson's paradox: A can win every slice and still lose overall. Compare slice by slice, or better, run both on one fixed test set with the same mix, then check the slices you care about.

Score the end-to-end answer and each step on its own: step 1 by similarity of the extracted text to the true text, step 2 by accuracy when it's given correct text.

Then a failure points at a step, not the whole system. Add slices such as file formats and layouts, include out-of-scope inputs, and log each step's output so failures can be traced.

Ask what decision the output drives. Approve or decline is binary, so use precision and recall at the operating threshold, set by error costs. How much, like a loss amount, is regression, judged by MAE or RMSE.

Often it's both: predict the probability of default and the loss if it happens, and combine them into expected value per decision. Agree the metric with stakeholders before modelling, and validate on a time-based split.

Use them when labels are scarce, delayed or blind to new attacks: isolation forests, autoencoder reconstruction error or graph clustering flag what's unusual, not what's known fraud.

Evaluate by proxy. Backtest on the labels you do have with precision at the top k, since investigators can review only so many. Send flagged and random cases to analysts to estimate precision and lift. Inject synthetic anomalies to check recall, and track alert volume and stability over time. Unusual isn't fraud, so feed confirmed cases into a supervised model.

It adds the same constant to every logit, a monotonic shift, so the order is unchanged and ranking metrics like AUC stay identical. At a fixed threshold the boundary moves: a positive shift labels more examples positive, so recall rises or holds and precision usually falls. Calibration breaks, since predicted probabilities no longer match observed rates. It's equivalent to moving the threshold, which is how prior correction after resampling works.

Validate out of time, only on examples whose labels have matured. With 3% positives, report PR-AUC, not accuracy.

Calibrate with Platt or isotonic regression on a recent held-out window, check reliability curves and log loss, and undo any resampling first.

Then act when expected value is positive: with gain G from a true positive and cost C from a false positive, the threshold is C / (C + G). Better still, sweep thresholds for expected profit with confidence intervals under capacity limits. Until labels arrive, monitor score distributions.