Cost per request interview questions

Input tokens × input price plus output tokens × output price. Output tokens cost more, roughly two to four times input across providers, per AI Engineering. At $2 and $10 per million, Sonnet 5.5's list prices (checked 21 October 2026), an illustrative 2,000-in, 500-out request spends 56% of its cost on the reply.

Divide the price per hour by tokens per hour: the book's $2 an hour at 100 tokens/s is about $5.56 per million output tokens. Batching raises throughput, so it cuts cost: at an illustrative $2 an hour, our Apple GPU's measured throughput gives $3.50 per million at batch 1 and $0.26 at batch 64, about 13 times cheaper. But bigger batches slow each user, and the KV cache caps the batch.

B. Track cost per useful answer: spend divided by answers that were on time and correct. Retrying until success takes 1 ÷ p attempts on average, so A costs $0.004 ÷ 0.5 = $0.008 per correct answer and B $0.006 ÷ 0.95 ≈ $0.0063. Lateness counts too: in lesson 1's CPU run, 83 of 120 requests met the SLO, so each on-time request cost 1.45 times the average.

Break-even requests per day = machine cost per day ÷ API cost per request, if one machine can serve that many. An illustrative $48-a-day machine against $0.002304 per request crosses at about 20,800 a day.

Left out: idle time (at 1,000 requests a day that machine costs $0.048 per request, 0.23% busy); quality, since our throughput is GPT-2 on a laptop, not a frontier model; engineering, which the book says can cost more than the API; and tokenizers, so compare per request. Re-run it as traffic grows.

Find where the tokens go first: output tokens cost more each, but long prompts can still dominate the bill.

Then cap and shorten outputs, cache the static prefix, trim retrieved context, route easy requests to a smaller model, run offline work through batch APIs, and cache whole answers only for exact repeats. Track cost per correct answer, so savings don't hide a quality drop.

When it passes your own eval set at your quality bar, not a leaderboard. Narrow, well-specified tasks like classification, extraction and routing often do; open-ended reasoning often doesn't. Compare cost per correct answer, include the cost of hosting it yourself, and consider a router: the small model first, escalating hard cases.

Route by difficulty and latency budget: easy, high-volume requests to the small model, hard ones to the large. The router can be rules, a small classifier, or a cascade that tries the cheap model and escalates when a check fails, which adds a second call's latency on escalations.

Set thresholds on a labelled set by plotting quality against cost per request, and watch p95, where cascades hurt.

Route on what each query needs. The on-device model takes device actions, short answers and private requests: fast, offline, no per-query cost. Open-ended, knowledge-heavy or long reasoning requests go to the server model.

The router is rules for clear intents, then a small on-device classifier scoring difficulty, plus escalation when the local answer's confidence is low. Ask before sending personal context off the device. Track quality per cost and latency: share escalated, quality against always-server on a labelled set, and tail latency. Retrain on misrouted cases.

Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry. Jobs queue per tenant and priority, with quotas against starvation.

Workers resolve the prompt version pinned in the request, route to a model by task and budget, and store results; clients poll or get a signed webhook. Control cost with token estimates at admission, budget caps, batch APIs for non-urgent work, prompt caching and per-tenant metering. Failures back off, then dead-letter.