Topic 5

From a score to a business number

An application exists to serve a business goal, so its quality scores need a meaning in business terms. AI Engineering (Ch. 4, "Step 2. Create an Evaluation Guideline", "Tie evaluation metrics to business metrics") asks: if your support chatbot's factual consistency is 80%, what does that mean for the business?

The answer might be that it's unusable for billing questions but fine for product recommendations or general feedback. Ideally, you map quality levels to business outcomes, for example: at this consistency we can automate this share of support requests, at a higher one, more. Then you know what improving a metric is worth before spending on it.

The book also recommends a usefulness threshold: the score below which the application isn't useful at all.

Predict first

The assistant's overall factual consistency is 90.4%, and your bar for automating a kind of question is 90%. Can it handle all the support traffic on its own?

Yes: 90.4% is above the barNot necessarily: some kinds of question may score below 90% even though the average is above itNo: nothing is ever automated fullyOnly if latency is low

How much support traffic can the assistant handle?

Illustrative traffic and scores · not from a real product

Usefulness bar: lowest factual consistency you'd automate90%

Traffic automated

70%

Left to people

30%

Billing30% of traffic · 82% ✗
Account20% of traffic · 90% ✓
Product how-to35% of traffic · 95% ✓
General feedback15% of traffic · 97% ✓

Overall factual consistency is 90.4%, a traffic-weighted average that hides the 82% on billing. A slice is automated only if its own score reaches the bar.

At a 90% bar, three of the four kinds pass and 70% of the traffic can be automated. Raise the bar to 95% and only product questions and general feedback pass: 50%. At 98%, nothing does. All of these numbers are illustrative; your own come from scoring your own traffic.

The business side matters too. The book notes that many products track stickiness, such as daily, weekly or monthly active users, or engagement, such as how long a visit lasts. It warns that chasing engagement alone can push a product towards addictive features or extreme content that's bad for its users.

Quick check

Why work out what a quality score means for the business before you invest in improving it?

So you know what moving the metric is worth, and which improvements are worth paying forSo the score looks betterBecause business metrics are easier to computeBecause quality scores can't be trusted