Topic 2

Same machine, different numbers

Fixing the workload isn't enough. The same measurement, repeated on the same machine on another day, comes out different. This module has the evidence: three scripts timed the same CPU prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught of GPT-2.

It is tempting to name a cause: lesson 1 ran on battery and lesson 2 on mains. We can't, because the methods also differ:

• Module 01 timed a forward pass alone, without the KV cacheKV cacheStored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.Where this is taught and without reading a tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught back. Its run did not record the power state.
• Lesson 1 timed the first token inside a streamed request, including choosing the token and reading it back, with each repeat also decoding 31 more tokens.
• Lesson 2 timed prefill with the cache and a token read back, after a sweep of decode batches up to 128.

Any of these, or none of them, could explain the gap. Isolating it would need runs that change one thing at a time.

What this means in practice: compare numbers from one run where you can, repeat runs before trusting a gap, and treat a difference smaller than your own run-to-run spread as no difference. That's why lessons 1 and 2 quote ratios.