Throughput
Work done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.
Interview questions about Throughput
- A fine-tuned BERT encoder is too slow and expensive in production. How would you optimize its latency and throughput, and how would you check quality held?
Profile first. Cut padding with dynamic padding and length bucketing, batch requests, and serve through an optimized runtime with fused kernels in FP16 or INT8.
- How does pipeline parallelism interact with GRPO rollouts, policy versions, throughput and staleness?
Pipeline parallelism splits layers into stages and streams microbatches through them, leaving a fill-and-drain bubble.
- You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?
Anything that stores tokens, because tokenizer mismatches fail silently: decoding IDs with the wrong tokenizer returns plausible garbage, not an error.
- Your assistant's semantic search over user content is slow and expensive. How do you optimise it: embedding choice, ANN indexing and sharding?
Embeddings: shortlist models on public retrieval benchmarks, then pick on your own queries; smaller dimensions or int8 vectors cut memory and latency.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?
Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each…
- Design a batched LLM inference service, assuming every batch takes 100 ms to run.
If a batch costs 100 ms whatever its size, throughput is batch size divided by 0.1 s, so bigger batches are free until memory runs out.
- Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?
Each step reads the weights once and uses them for every sequence; only each sequence's KV cache is extra.
- Your training GPUs sit partly idle waiting for data. How do you confirm that, and how would you design a data-preparation pipeline that keeps them fed?
First prove it: a profiler shows each step waiting on the next batch, and GPU utilisation dips in a sawtooth.