You're shipping a classifier for 7-day purchase with 3% positives, a 10-day label delay and asymmetric business values. How do you calibrate it and pick the threshold?

Validate out of time, only on examples whose labels have matured. With 3% positives, report PR-AUC, not accuracy.

Calibrate with Platt or isotonic regression on a recent held-out window, check reliability curves and log loss, and undo any resampling first.

Then act when expected value is positive: with gain G from a true positive and cost C from a false positive, the threshold is C / (C + G). Better still, sweep thresholds for expected profit with confidence intervals under capacity limits. Until labels arrive, monitor score distributions.