Topic 5
Static, dynamic and continuous batching
Requests don't arrive together, so a server has to decide when to start a batch. AI Engineering (Ch. 9, "Batching") describes three ways:
- Static batching waits until the batch is full. The first request waits for the last one to arrive.
- Dynamic batching starts when the batch is full or a time window runs out, whichever comes first. Waiting is capped, but batches may run part-empty.
- Continuous batching lets a finished reply leave straight away and puts a waiting request in its seat. It comes from Orca (Yu et al., OSDI 2022), which calls it iteration-level scheduling.
With the first two, a batch finishes together: a 10-token reply waits for its 1,000-token neighbour.
Scroll sideways to see the whole diagram
Two requests are decoded together: A needs 8 more tokens, B needs 64.
Text version
- Two requests are decoded together: A needs 8 more tokens, B needs 64.
- In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
- In continuous batching A is returned as soon as its last token is generated.
- Its seat is filled by a waiting request at the very next step, so the batch stays full.
- Orca calls this iteration-level scheduling: the batch is decided again before every step.
After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.
Quick check
The next page puts numbers on this.