Topic 3

Batching shares the read

If decode spends its time reading weights, the obvious trick is to use each read for more than one sequence. Decode 8 conversations together and the 498 MB of weights is read once per step for all 8; only each conversation's KV cacheKV cacheStored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.Where this is taught is extra.

ThroughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught keeps rising with the batch, then stops. On the CPU it peaked at 1,183 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught/s at batch 64 and fell to 972 at batch 128. That last point is just past the CPU's ridge in operations per byteArithmetic intensityOperations done per byte read from memory. Low intensity means bandwidth is the limit; high intensity means compute is.Where this is taught, but it reaches only about 242 of the 1,953 billion operations per second the CPU managed on a plain matrix multiply, so it is not simply hitting the compute roof. We didn't measure what does limit it. The GPU's step at batch 128 jumped to 536 ms in this run; we did not re-run or investigate it.

Each user still pays: the step got longer, so each conversation's tokens arrive more slowly (lesson 1's throughput page).

A note on repeatability: this run's CPU steps at batch 1, 4 and 16 were 9.1, 14.0 and 19.9 ms. Lesson 1, using the same code in a separate run, measured 8.0, 13.5 and 18.5 ms. The same machine gives numbers up to 14% apart between runs, which is why we quote ratios.