# Latency, throughput and goodput interview questions

> 17 latency, throughput and goodput interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/latency-and-throughput
Last updated: 2026-10-23

17 questions: 4 junior, 7 mid-level and 6 senior. Taught in [Measuring Inference: Latency, Throughput and Goodput](https://llmdojo.dev/learn/inference-optimization/measuring-inference).

## [A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?](https://llmdojo.dev/interview/q/a-chat-reply-streams-in-token-by-token-what-two-waits-does)

Level: Junior

Time to first token (TTFT), **mostly prefill**, and total latency, **mostly decode**. Streaming makes TTFT **the visible wait**: on our test laptop's CPU, GPT-2 small's first token arrived at 67.5 ms, the last at 633 ms. A fast reader takes about 120 ms per token, so tokens needn't arrive much faster than that.

## [How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?](https://llmdojo.dev/interview/q/how-do-you-get-ttft-inter-token-latency-tpot-and-total-latency)

Level: Junior

Record **one timestamp per token**, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total latency is tN − t0, about TTFT + TPOT × (N − 1).

**Diagram: Where each latency number comes from**

1. Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
2. A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
3. Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
4. Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
5. The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.

Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.

**Keep every timestamp**: our CPU stream's median gap was 8.6 ms, the slowest 11.5 ms.

## [Why report p50 and p99 latency rather than the mean?](https://llmdojo.dev/interview/q/why-report-p50-and-p99-latency-rather-than-the-mean)

Level: Junior

**Latency is a distribution**, and the **mean hides the tail**. The same request sent 200 times on our laptop CPU had a p50 of 61.3 ms, a p99 of 112.8 ms and a slowest run of 176.2 ms. **Users meet the tail**: send 100 messages a day and you hit the p99 about daily.

## [You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?](https://llmdojo.dev/interview/q/you-time-a-gpu-request-by-reading-the-clock-right-after-calling)

Level: Mid-level

**GPU calls are asynchronous**: they return before the GPU finishes, so I timed the queuing, not the work. Wait for a result, or **synchronise the device**, before stopping the clock.

Two more traps give no error: **skipping warm-up**, since first calls compile code and fill caches; and where the clock sits, since server-side timing leaves out the network and queue users feel.

## [Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.](https://llmdojo.dev/interview/q/server-logs-show-a-ttft-of-80-ms-yet-users-of-your)

Level: Mid-level

TTFT counts the first generated token; the user waits for the **first visible one**, after every hidden plan or reasoning token, plus any tool calls. Some teams call that time to publish. In our CPU stream with the first 48 tokens hidden, the first visible token came at 504.2 ms instead of 67.5 ms. **Measure what the user sees**, alongside TTFT.

## [Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?](https://llmdojo.dev/interview/q/your-team-wants-to-raise-the-batch-size-to-cut-serving-cost)

Level: Senior

Batching **raises total throughput** and **slows each user**. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each user dropped to 54.

So I'd decide on **goodput**: requests per minute meeting the SLO, say first token within 100 ms and at most 10 ms per token. Throughput also counts requests users gave up on, so goodput can fall while it rises. On a 120-request CPU workload, 83 met that SLO.

## [Explain how a server like vLLM runs a decoder-only model efficiently.](https://llmdojo.dev/interview/q/explain-how-a-server-like-vllm-runs-a-decoder-only-model-efficiently)

Level: Senior

A decoder-only model runs **prefill**, then one decode step per token, and decode is memory-bound, so the win is **batching many requests**.

**Diagram: Request-level versus continuous batching**

1. Two requests are decoded together: A needs 8 more tokens, B needs 64.
2. In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
3. In continuous batching A is returned as soon as its last token is generated.
4. Its seat is filled by a waiting request at the very next step, so the batch stays full.
5. Orca calls this iteration-level scheduling: the batch is decided again before every step.

After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.

**Continuous batching** adds and removes requests at every step instead of waiting for a whole batch to finish. **PagedAttention** stores each request's KV cache in fixed-size blocks, like memory pages, so nothing is reserved for the maximum length and wasted: more requests fit, and shared prefixes can share blocks.

## [Your AI chatbot takes too long to respond. What do you do?](https://llmdojo.dev/interview/q/your-ai-chatbot-takes-too-long-to-respond-what-do-you-do)

Level: Mid-level

First **measure which wait** it is: time to first token or total time, at p50 and p99.

**Diagram: Where each latency number comes from**

1. Start the clock when the request leaves the client. Measured from the client, everything that follows counts, which is what a user feels.
2. A hosted service adds network time and time waiting in a queue behind other requests. Our runs are on one laptop with nothing else running, so this is zero.
3. Prefill reads the prompt and produces the first token. The time from t0 to its arrival is the time to first token (TTFT).
4. Each decode step produces one more token. Record every arrival: the gaps between them are the inter-token latencies, and their average is the time per output token (TPOT).
5. The last arrival gives the total latency. A record of arrival times is enough to compute all three numbers afterwards.

Standard streaming measurement. Our script (scripts/generate-latency-data.py) takes t0 before the model call and each token's time after reading it back to the CPU.

Slow first token: shorten the prompt, **cache the static prefix**, cut queueing. Slow total: **cap output length**, use a smaller or quantized model, check batching. Always **stream**, so users read while it writes, and fix the biggest measured part first.

## [Design request batching for GPU inference, including scheduling, autoscaling and routing.](https://llmdojo.dev/interview/q/design-request-batching-for-gpu-inference-including-scheduling-autoscaling-and-routing)

Level: Senior

Batch at the **iteration level**: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long ones. The scheduler admits work while **KV-cache memory** allows, using paged allocation, and chunks long prefills so they don't stall decodes.

**Diagram: Request-level versus continuous batching**

1. Two requests are decoded together: A needs 8 more tokens, B needs 64.
2. In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
3. In continuous batching A is returned as soon as its last token is generated.
4. Its seat is filled by a waiting request at the very next step, so the batch stays full.
5. Orca calls this iteration-level scheduling: the batch is decided again before every step.

After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.

Route by queue depth and free cache, preferring a replica that holds the prompt prefix. Autoscale on queue length and time to first token, not GPU utilisation, keep warm spares because loading weights is slow, and shed batch traffic first under overload.

## [Design a batched LLM inference service, assuming every batch takes 100 ms to run.](https://llmdojo.dev/interview/q/design-a-batched-llm-inference-service-assuming-every-batch-takes-100-ms)

Level: Mid-level

If a batch costs 100 ms whatever its size, throughput is **batch size divided by 0.1 s**, so bigger batches are free until memory runs out. Latency is the wait for the next batch plus 100 ms. A batcher fires when the batch fills or a timeout expires. **Bound the queue** and reject when full, so latency stays predictable. For long generations, batch per decode step instead.

## [Design the routing and scheduling layer in front of an inference fleet that serves many tenants.](https://llmdojo.dev/interview/q/design-the-routing-and-scheduling-layer-in-front-of-an-inference-fleet)

Level: Senior

Three jobs. **Admission**: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch. **Fair scheduling** inside each replica: weighted fair queuing across tenants, so one heavy tenant can't fill every batch slot. **Routing**: choose replicas by load and prefix-cache affinity, so repeated system prompts hit warm KV cache, spilling over when a replica runs hot.

**Diagram: Where a prefix match breaks**

1. Put what never changes first. The cache can reuse everything up to the first token that differs.
2. Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
3. This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.

Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.

Track time to first token and per-token latency per tenant against SLOs, and under overload shed the lowest priority first with retry-after errors.

## [How would you optimise large-scale training and inference on GPUs?](https://llmdojo.dev/interview/q/how-would-you-optimise-large-scale-training-and-inference-on-gpus)

Level: Senior

Profile first, then attack the **binding limit**. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade compute for memory, ZeRO or FSDP sharding, and parallelism laid out to the topology, tensor parallel inside NVLink, pipeline and data across nodes, with communication overlapped. Track **MFU**.

Inference: **decode is memory-bound**, so cut bytes with quantised weights and KV cache and GQA, raise batch size with paged KV and continuous batching, use speculative decoding for latency, and add tensor parallelism only as far as latency needs.

## [Design inference serving for a model with 100 million daily users. Give back-of-the-envelope QPS and memory for weights and caches, and explain how you'd control tail latency.](https://llmdojo.dev/interview/q/design-inference-serving-for-a-model-with-100-million-daily-users-give)

Level: Mid-level

Assume 100 million users each send 10 requests a day: about **11,600 QPS** average, peaks perhaps 2 to 3 times higher. Weights are parameters times bytes, so a 70B model in 16-bit is 140 GB, sharded across GPUs. KV cache per token is 2 times layers times KV heads times head dimension times bytes, times context and concurrency.

**Diagram: Request-level versus continuous batching**

1. Two requests are decoded together: A needs 8 more tokens, B needs 64.
2. In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
3. In continuous batching A is returned as soon as its last token is generated.
4. Its seat is filled by a waiting request at the very next step, so the batch stays full.
5. Orca calls this iteration-level scheduling: the batch is decided again before every step.

After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.

For tail latency: **continuous batching**, admission control, split prefill and decode, autoscale on queue depth.

## [Design an LLM-powered customer service chat that keeps working when the external LLM provider is slow, fails or returns bad output.](https://llmdojo.dev/interview/q/design-an-llm-powered-customer-service-chat-that-keeps-working-when-the)

Level: Mid-level

Assume the provider will be slow, down or wrong. Set **timeouts** on time to first token, retry with backoff and jitter only on safe errors, and use a circuit breaker to **fail over** to another provider or smaller model. Validate each output against a schema and policy checks before showing it; on failure, retry once, then use a scripted reply or human handoff. Keep state server-side so failover resumes cleanly.

## [Design an AI gateway in front of several external LLM providers, with routing and fallback, quotas, retries, circuit breakers and privacy-preserving logging.](https://llmdojo.dev/interview/q/design-an-ai-gateway-in-front-of-several-external-llm-providers-with)

Level: Mid-level

One API; adapters normalize each provider's schema and streaming. Route by model alias, cost and region; fall back only to models **evaluated as equivalent**.

Quotas: token buckets per tenant, counting requests and tokens. Retry only timeouts, 429s and 5xx, with jittered backoff, honouring Retry-After, under a retry budget. A **circuit breaker** per provider opens on error rate, then sends probes.

Log metadata always: tokens, latency, cost, status. Bodies are **redacted** or skipped, with short retention.

## [A fine-tuned BERT encoder is too slow and expensive in production. How would you optimize its latency and throughput, and how would you check quality held?](https://llmdojo.dev/interview/q/a-fine-tuned-bert-encoder-is-too-slow-and-expensive-in-production)

Level: Junior

Profile first. Cut padding with **dynamic padding** and length bucketing, batch requests, and serve through an optimized runtime with fused kernels in FP16 or INT8. If that's not enough, **distill** into a smaller model.

Check quality on a held-out set: accuracy and F1 overall and per slice, plus **agreement** with the original. Then shadow traffic.

## [Design a production ML inference pipeline covering request routing to models, artifact versioning and deployment, and online feature retrieval.](https://llmdojo.dev/interview/q/design-a-production-ml-inference-pipeline-covering-request-routing-to-models-artifact)

Level: Senior

A router consults a **model registry** that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.

Artifacts are immutable and versioned together: weights, preprocessing and feature schema, with lineage to data and code. CI checks offline metrics, schema compatibility and latency before promotion; canaries **roll back automatically** on SLO breaches.

Features come from an online store by entity key, with timeouts and defaults, using **the same transforms** as training; log what was served. Batch dynamically and autoscale on queue depth.

## Related
- [All interview questions](https://llmdojo.dev/interview)