# Cost per request interview questions > 9 cost per request interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/cost Last updated: 2026-10-23 9 questions: 1 junior, 5 mid-level and 3 senior. Taught in [Cost per Request and per Useful Answer](https://llmdojo.dev/learn/inference-optimization/cost-per-request). ## [How is an API request priced, and why is the reply often most of the bill?](https://llmdojo.dev/interview/q/how-is-an-api-request-priced-and-why-is-the-reply-often) Level: Junior Input tokens × input price plus output tokens × output price. **Output tokens cost more**, roughly two to four times input across providers, per AI Engineering. At $2 and $10 per million, Sonnet 5.5's list prices (checked 21 October 2026), an illustrative 2,000-in, 500-out request spends **56%** of its cost on the reply. **Diagram: Where an API request's cost comes from** 1. A request has two token counts: the prompt you send and the reply the model writes. 2. Input tokens are billed at the input price: 2,000 tokens at $2 per million is $0.004. 3. Output tokens have their own, higher price: 500 tokens at $10 per million is $0.005. 4. Add the two. The reply is a fifth of the tokens but more than half the bill, so trimming replies often saves the most. Claude Sonnet 5.5 list prices, $2 input / $10 output per million tokens: platform.claude.com/docs/en/about-claude/pricing, fetched 2026-10-21. Token counts illustrative. ## [How do you turn a machine's hourly price into a cost per token, and how much can batching change it?](https://llmdojo.dev/interview/q/how-do-you-turn-a-machine-s-hourly-price-into-a-cost) Level: Mid-level Divide the price per hour by **tokens per hour**: the book's $2 an hour at 100 tokens/s is about $5.56 per million output tokens. **Batching raises throughput**, so it cuts cost: at an illustrative $2 an hour, our Apple GPU's measured throughput gives $3.50 per million at batch 1 and $0.26 at batch 64, **about 13 times cheaper**. But bigger batches slow each user, and the KV cache caps the batch. ## [(Illustrative numbers.) Model A costs $0.004 per request and is correct 50% of the time; model B costs $0.006 and is correct 95% of the time. Which is cheaper, and what should you track?](https://llmdojo.dev/interview/q/illustrative-numbers-model-a-costs-0-004-per-request-and-is-correct) Level: Mid-level B. Track **cost per useful answer**: spend divided by answers that were **on time and correct**. Retrying until success takes 1 ÷ p attempts on average, so A costs $0.004 ÷ 0.5 = $0.008 per correct answer and B $0.006 ÷ 0.95 ≈ $0.0063. Lateness counts too: in lesson 1's CPU run, 83 of 120 requests met the SLO, so each on-time request cost 1.45 times the average. ## [You're deciding between a model API and self-hosting. How do you find the break-even, and what does the simple calculation leave out?](https://llmdojo.dev/interview/q/you-re-deciding-between-a-model-api-and-self-hosting-how-do) Level: Senior Break-even requests per day = **machine cost per day** ÷ **API cost per request**, if one machine can serve that many. An illustrative $48-a-day machine against $0.002304 per request crosses at about 20,800 a day. Left out: **idle time** (at 1,000 requests a day that machine costs $0.048 per request, 0.23% busy); quality, since our throughput is GPT-2 on a laptop, not a frontier model; engineering, which the book says can cost more than the API; and tokenizers, so compare per request. Re-run it as traffic grows. ## [How do you cut token costs at scale?](https://llmdojo.dev/interview/q/how-do-you-cut-token-costs-at-scale) Level: Mid-level Find where the tokens go first: **output tokens cost more each**, but long prompts can still dominate the bill. **Diagram: Where an API request's cost comes from** 1. A request has two token counts: the prompt you send and the reply the model writes. 2. Input tokens are billed at the input price: 2,000 tokens at $2 per million is $0.004. 3. Output tokens have their own, higher price: 500 tokens at $10 per million is $0.005. 4. Add the two. The reply is a fifth of the tokens but more than half the bill, so trimming replies often saves the most. Claude Sonnet 5.5 list prices, $2 input / $10 output per million tokens: platform.claude.com/docs/en/about-claude/pricing, fetched 2026-10-21. Token counts illustrative. Then cap and shorten outputs, **cache the static prefix**, trim retrieved context, route easy requests to a **smaller model**, run offline work through batch APIs, and cache whole answers only for exact repeats. Track cost per correct answer, so savings don't hide a quality drop. ## [When is a small open model good enough, rather than a frontier model?](https://llmdojo.dev/interview/q/when-is-a-small-open-model-good-enough-rather-than-a-frontier) Level: Mid-level When it passes **your own eval set** at your quality bar, not a leaderboard. Narrow, well-specified tasks like classification, extraction and routing often do; open-ended reasoning often doesn't. Compare **cost per correct answer**, include the cost of hosting it yourself, and consider a **router**: the small model first, escalating hard cases. ## [When do you send inference to a fast model and when to a cheap one? Walk me through the cost and latency trade-offs.](https://llmdojo.dev/interview/q/when-do-you-send-inference-to-a-fast-model-and-when-to) Level: Mid-level Route by **difficulty and latency budget**: easy, high-volume requests to the small model, hard ones to the large. The router can be rules, a small classifier, or a cascade that tries the cheap model and **escalates** when a check fails, which adds a second call's latency on escalations. **Diagram: Where an API request's cost comes from** 1. A request has two token counts: the prompt you send and the reply the model writes. 2. Input tokens are billed at the input price: 2,000 tokens at $2 per million is $0.004. 3. Output tokens have their own, higher price: 500 tokens at $10 per million is $0.005. 4. Add the two. The reply is a fifth of the tokens but more than half the bill, so trimming replies often saves the most. Claude Sonnet 5.5 list prices, $2 input / $10 output per million tokens: platform.claude.com/docs/en/about-claude/pricing, fetched 2026-10-21. Token counts illustrative. Set thresholds on a labelled set by plotting quality against cost per request, and watch p95, where cascades hurt. ## [Design the routing that decides whether a voice assistant query is answered by the on-device model or sent to a large server LLM.](https://llmdojo.dev/interview/q/design-the-routing-that-decides-whether-a-voice-assistant-query-is-answered) Level: Senior Route on what each query needs. The **on-device model** takes device actions, short answers and private requests: fast, offline, no per-query cost. Open-ended, knowledge-heavy or long reasoning requests go to the server model. The router is rules for clear intents, then a small on-device classifier scoring difficulty, plus escalation when the local answer's confidence is low. Ask before sending personal context off the device. Track **quality per cost and latency**: share escalated, quality against always-server on a labelled set, and tail latency. Retrain on misrouted cases. ## [Design a backend where clients submit LLM prompt jobs and poll or receive callbacks, with queueing, model routing, prompt versioning, idempotency and cost control.](https://llmdojo.dev/interview/q/design-a-backend-where-clients-submit-llm-prompt-jobs-and-poll-or) Level: Senior Clients submit a job with an **idempotency key**; the API stores it durably and returns a job id, the same id for any retry. Jobs queue per tenant and priority, with quotas against starvation. Workers resolve the **prompt version** pinned in the request, route to a model by task and budget, and store results; clients poll or get a signed webhook. Control cost with token estimates at admission, **budget caps**, batch APIs for non-urgent work, prompt caching and per-tenant metering. Failures back off, then dead-letter. ## Related - [All interview questions](https://llmdojo.dev/interview)