Topic 6

Simulating the strategies

Here are the three strategies on the same workload. The simulator replays 200 requests, timed with the decode and prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught times measured on this laptop. Switch the strategy and watch the timeline: the light part of each bar is waiting, and the reply starts at the tick.

With the defaults (Apple GPU, 4 requests per second, at most 8 at once), continuous batchingContinuous batchingBatching that re-decides the batch before every decode step: finished replies leave at once and waiting requests take their seats.Where this is taught produces 343 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught per second and the median first token arrives 22 ms after the request. Static batches of 8 produce 299 tokens per second, but the median request waits 5.3 s for its first token. One at a time can't keep up at all: 154 tokens per second, and the queue keeps growing, so the median first token comes after 33.7 s.

What to keep

• Decode reads every weight for every step: it is bandwidth-boundMemory bandwidth-boundWork whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.Where this is taught, at about half an operation per byte.
• Prefill reuses each read for every prompt token: it is compute-boundCompute-boundWork whose time is set by how much arithmetic it needs; a chip with more operations per second speeds it up.Where this is taught.
• Batching shares the weight read, so throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught rises until something else limits it. Memory for the KV cacheKV cacheStored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.Where this is taught caps the batch.
• How you form batches matters as much as their size: continuous batching keeps them full without making short replies wait.

Next: the cheapest way to read fewer bytes is to store each weight in fewer of them. Lesson 3 covers precision and quantizationQuantizationStoring a model's numbers in a format with fewer bits, such as FP16, INT8 or INT4, to cut memory and the bytes each step reads. Strictly it means converting to integers, but in practice it covers any precision reduction.Where this is taught.