Latency, throughput and goodput interview questions
17 questions with model answers
- junior 4
- mid 7
- senior 6
17 latency, throughput and goodput questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 4 junior, 7 mid-level and 6 senior. The lesson Measuring Inference: Latency, Throughput and Goodput teaches the ideas behind them.Updated
Model answer
Time to first token (TTFT), mostly prefill, and total latency, mostly decode. Streaming makes TTFT the visible wait: on our test laptop's CPU, GPT-2 small's first token arrived at 67.5 ms, the last at 633 ms. A fast reader takes about 120 ms per token, so tokens needn't arrive much faster than that.
Model answer
Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total latency is tN − t0, about TTFT + TPOT × (N − 1).
Scroll sideways to see the whole diagram
Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
Text version
- Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
- A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
- Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
- Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
- The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.
Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.
Keep every timestamp: our CPU stream's median gap was 8.6 ms, the slowest 11.5 ms.
Model answer
Latency is a distribution, and the mean hides the tail. The same request sent 200 times on our laptop CPU had a p50 of 61.3 ms, a p99 of 112.8 ms and a slowest run of 176.2 ms. Users meet the tail: send 100 messages a day and you hit the p99 about daily.
Model answer
GPU calls are asynchronous: they return before the GPU finishes, so I timed the queuing, not the work. Wait for a result, or synchronise the device, before stopping the clock.
Two more traps give no error: skipping warm-up, since first calls compile code and fill caches; and where the clock sits, since server-side timing leaves out the network and queue users feel.
Model answer
TTFT counts the first generated token; the user waits for the first visible one, after every hidden plan or reasoning token, plus any tool calls. Some teams call that time to publish. In our CPU stream with the first 48 tokens hidden, the first visible token came at 504.2 ms instead of 67.5 ms. Measure what the user sees, alongside TTFT.
Model answer
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each user dropped to 54.
So I'd decide on goodput: requests per minute meeting the SLO, say first token within 100 ms and at most 10 ms per token. Throughput also counts requests users gave up on, so goodput can fall while it rises. On a 120-request CPU workload, 83 met that SLO.
Model answer
A decoder-only model runs prefill, then one decode step per token, and decode is memory-bound, so the win is batching many requests.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
Continuous batching adds and removes requests at every step instead of waiting for a whole batch to finish. PagedAttention stores each request's KV cache in fixed-size blocks, like memory pages, so nothing is reserved for the maximum length and wasted: more requests fit, and shared prefixes can share blocks.
Model answer
First measure which wait it is: time to first token or total time, at p50 and p99.
Scroll sideways to see the whole diagram
Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
Text version
- Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
- A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
- Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
- Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
- The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.
Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.
Slow first token: shorten the prompt, cache the static prefix, cut queueing. Slow total: cap output length, use a smaller or quantized model, check batching. Always stream, so users read while it writes, and fix the biggest measured part first.
Model answer
Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long ones. The scheduler admits work while KV-cache memory allows, using paged allocation, and chunks long prefills so they don't stall decodes.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
Route by queue depth and free cache, preferring a replica that holds the prompt prefix. Autoscale on queue length and time to first token, not GPU utilisation, keep warm spares because loading weights is slow, and shed batch traffic first under overload.
Model answer
If a batch costs 100 ms whatever its size, throughput is batch size divided by 0.1 s, so bigger batches are free until memory runs out. Latency is the wait for the next batch plus 100 ms. A batcher fires when the batch fills or a timeout expires. Bound the queue and reject when full, so latency stays predictable. For long generations, batch per decode step instead.
Model answer
Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch. Fair scheduling inside each replica: weighted fair queuing across tenants, so one heavy tenant can't fill every batch slot. Routing: choose replicas by load and prefix-cache affinity, so repeated system prompts hit warm KV cache, spilling over when a replica runs hot.
Scroll sideways to see the whole diagram
Put what never changes first. The cache can reuse everything up to the first token that differs.
Text version
- Put what never changes first. The cache can reuse everything up to the first token that differs.
- Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
- This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.
Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.
Track time to first token and per-token latency per tenant against SLOs, and under overload shed the lowest priority first with retry-after errors.
Model answer
Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade compute for memory, ZeRO or FSDP sharding, and parallelism laid out to the topology, tensor parallel inside NVLink, pipeline and data across nodes, with communication overlapped. Track MFU.
Inference: decode is memory-bound, so cut bytes with quantised weights and KV cache and GQA, raise batch size with paged KV and continuous batching, use speculative decoding for latency, and add tensor parallelism only as far as latency needs.
Model answer
Assume 100 million users each send 10 requests a day: about 11,600 QPS average, peaks perhaps 2 to 3 times higher. Weights are parameters times bytes, so a 70B model in 16-bit is 140 GB, sharded across GPUs. KV cache per token is 2 times layers times KV heads times head dimension times bytes, times context and concurrency.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
For tail latency: continuous batching, admission control, split prefill and decode, autoscale on queue depth.
Model answer
Assume the provider will be slow, down or wrong. Set timeouts on time to first token, retry with backoff and jitter only on safe errors, and use a circuit breaker to fail over to another provider or smaller model. Validate each output against a schema and policy checks before showing it; on failure, retry once, then use a scripted reply or human handoff. Keep state server-side so failover resumes cleanly.
Model answer
One API; adapters normalize each provider's schema and streaming. Route by model alias, cost and region; fall back only to models evaluated as equivalent.
Quotas: token buckets per tenant, counting requests and tokens. Retry only timeouts, 429s and 5xx, with jittered backoff, honouring Retry-After, under a retry budget. A circuit breaker per provider opens on error rate, then sends probes.
Log metadata always: tokens, latency, cost, status. Bodies are redacted or skipped, with short retention.
Model answer
Profile first. Cut padding with dynamic padding and length bucketing, batch requests, and serve through an optimized runtime with fused kernels in FP16 or INT8. If that's not enough, distill into a smaller model.
Check quality on a held-out set: accuracy and F1 overall and per slice, plus agreement with the original. Then shadow traffic.
Model answer
A router consults a model registry that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.
Artifacts are immutable and versioned together: weights, preprocessing and feature schema, with lineage to data and code. CI checks offline metrics, schema compatibility and latency before promotion; canaries roll back automatically on SLO breaches.
Features come from an online store by entity key, with timeouts and defaults, using the same transforms as training; log what was served. Batch dynamically and autoscale on queue depth.