# Throughput

> Throughput: Work done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.

Canonical: https://llmdojo.dev/glossary/throughput

Work done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.

Learn it in the lesson: [Throughput: everyone vs each user](https://llmdojo.dev/learn/inference-optimization/measuring-inference/throughput) (Inference and Model Optimization · Measuring Inference: Latency, Throughput and Goodput).

## Interview questions about Throughput

- [A fine-tuned BERT encoder is too slow and expensive in production. How would you optimize its latency and throughput, and how would you check quality held?](https://llmdojo.dev/interview/q/a-fine-tuned-bert-encoder-is-too-slow-and-expensive-in-production)
- [How does pipeline parallelism interact with GRPO rollouts, policy versions, throughput and staleness?](https://llmdojo.dev/interview/q/how-does-pipeline-parallelism-interact-with-grpo-rollouts-policy-versions-throughput-and)
- [You're moving a product from a model that uses cl100k_base to one that uses o200k_base. Nothing crashes in staging. What could still be broken, and what goes into the migration plan?](https://llmdojo.dev/interview/q/you-re-moving-a-product-from-a-model-that-uses-cl100k-base)
- [Your assistant's semantic search over user content is slow and expensive. How do you optimise it: embedding choice, ANN indexing and sharding?](https://llmdojo.dev/interview/q/your-assistant-s-semantic-search-over-user-content-is-slow-and-expensive)
- [Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?](https://llmdojo.dev/interview/q/your-team-wants-to-raise-the-batch-size-to-cut-serving-cost)
- [Design a batched LLM inference service, assuming every batch takes 100 ms to run.](https://llmdojo.dev/interview/q/design-a-batched-llm-inference-service-assuming-every-batch-takes-100-ms)
- [Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?](https://llmdojo.dev/interview/q/why-does-decoding-a-batch-of-8-cost-far-less-than-8)
- [Your training GPUs sit partly idle waiting for data. How do you confirm that, and how would you design a data-preparation pipeline that keeps them fed?](https://llmdojo.dev/interview/q/your-training-gpus-sit-partly-idle-waiting-for-data-how-do-you)

## Related
- [Time to first token (TTFT)](https://llmdojo.dev/glossary/ttft)
- [Time per output token (TPOT)](https://llmdojo.dev/glossary/tpot)
- [Inter-token latency](https://llmdojo.dev/glossary/itl)
- [Percentile (p50, p99)](https://llmdojo.dev/glossary/percentile)
- [Goodput](https://llmdojo.dev/glossary/goodput)
- [Service level objective (SLO)](https://llmdojo.dev/glossary/slo)
- [Time to publish](https://llmdojo.dev/glossary/time-to-publish)
- [Glossary of every term](https://llmdojo.dev/glossary)