# Continuous batching

> Continuous batching: Batching that re-decides the batch before every decode step: finished replies leave at once and waiting requests take their seats.

Canonical: https://llmdojo.dev/glossary/continuous-batching

Also called in-flight batching.

Batching that re-decides the batch before every decode step: finished replies leave at once and waiting requests take their seats.

Learn it in the lesson: [Static, dynamic and continuous batching](https://llmdojo.dev/learn/inference-optimization/where-the-time-goes/batching-strategies) (Inference and Model Optimization · Where the Time Goes: Memory, Compute and Batching).

## Interview questions about Continuous batching

- [In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?](https://llmdojo.dev/interview/q/in-the-capstone-continuous-batching-with-at-most-8-seats-gives-a)
- [Explain how a server like vLLM runs a decoder-only model efficiently.](https://llmdojo.dev/interview/q/explain-how-a-server-like-vllm-runs-a-decoder-only-model-efficiently)
- [Design request batching for GPU inference, including scheduling, autoscaling and routing.](https://llmdojo.dev/interview/q/design-request-batching-for-gpu-inference-including-scheduling-autoscaling-and-routing)
- [How would you optimise large-scale training and inference on GPUs?](https://llmdojo.dev/interview/q/how-would-you-optimise-large-scale-training-and-inference-on-gpus)
- [Design inference serving for a model with 100 million daily users. Give back-of-the-envelope QPS and memory for weights and caches, and explain how you'd control tail latency.](https://llmdojo.dev/interview/q/design-inference-serving-for-a-model-with-100-million-daily-users-give)
- [Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?](https://llmdojo.dev/interview/q/your-chat-app-s-replies-range-from-8-to-256-tokens-which)
- [You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?](https://llmdojo.dev/interview/q/you-need-to-deploy-multimodal-models-under-tight-gpu-and-latency-limits)

## Related
- [Compute-bound](https://llmdojo.dev/glossary/compute-bound)
- [Memory bandwidth-bound](https://llmdojo.dev/glossary/bandwidth-bound)
- [Arithmetic intensity](https://llmdojo.dev/glossary/arithmetic-intensity)
- [Roofline](https://llmdojo.dev/glossary/roofline)
- [Model bandwidth utilization (MBU)](https://llmdojo.dev/glossary/mbu)
- [Glossary of every term](https://llmdojo.dev/glossary)