Continuous batching
Also called in-flight batching
Batching that re-decides the batch before every decode step: finished replies leave at once and waiting requests take their seats.
Interview questions about Continuous batching
- In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?
It has no headroom. Capacity, the highest arrival rate still within 300 ms at p90, is 6 per second against a target of 9: eight seats fill as traffic grows…
- Explain how a server like vLLM runs a decoder-only model efficiently.
A decoder-only model runs prefill, then one decode step per token, and decode is memory-bound, so the win is batching many requests.
- Design request batching for GPU inference, including scheduling, autoscaling and routing.
Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long…
- How would you optimise large-scale training and inference on GPUs?
Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade…
- Design inference serving for a model with 100 million daily users. Give back-of-the-envelope QPS and memory for weights and caches, and explain how you'd control tail latency.
Assume 100 million users each send 10 requests a day: about 11,600 QPS average, peaks perhaps 2 to 3 times higher.
- Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?
Continuous batching. Static batching waits for a full batch and dynamic caps the wait, but both finish a batch together, so a short reply waits for its…
- You need to deploy multimodal models under tight GPU and latency limits, and also support fast retrieval over video. How do you approach it?
Profile first: preprocessing, vision encoder, prefill and decode each take their share.