Topic 4

Memory sets the batch

Bigger batches need room. Every conversation in the batch keeps its own KV cacheKV cacheStored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.Where this is taught in accelerator memory, next to one copy of the weights:

weights + KV cache per conversation × conversations ≤ memory

The KV cache formula is the one from Module 01: 2 × layers × heads × head size × tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught × bytes per number.

Try halving the tokens per conversation in the estimator: about twice as many conversations fit. Switching to 4-byte numbers costs more than half of them, because the weights double as well as the cache. Real servers fit fewer than this estimate, because activations, the framework itself and fragmentation all take memory.

This is where the book's two meanings of memory-boundMemory bandwidth-boundWork whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.Where this is taught meet: bandwidth decides how fast each step runs, capacity decides how many sequences can share it.