Topic 3

Hardware speed-ups depend on the workload

"The GPU is twice as fast" is never true in general. It is true for one phase, one prompt length and one batch size, measured in one run.

Read down the last column. Each ratio divides a CPU time by the Apple GPU time from the same run, so both sides share that run's conditions:

• PrefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught (the first tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught): lesson 1 found the GPU 1.6× faster at 16 tokens and 2.3× at 960. Lesson 2 found 2.4× at 16 and 2.3× at 960. The runs disagree on short prompts, because the CPU's 16-token time differed between them (12.8 vs 21.3 ms): the run-to-run problem from the previous page, now inside a ratio.
• Decode at batch 1: about 1.4× in both runs, well below the prefill ratio. That fits lesson 2: decode is limited by memory bandwidth, and the two measured copy ceilings (122.3 and 129.6 GB/s) were close.
• Decode at larger batches: 1.8× at batch 64. At batch 128 the GPU was slower, 0.25×, in a single run we did not repeat or investigate.

AI Engineering (Ch. 9, "AI Accelerators", sidebar "Selecting Accelerators") puts the choice as three questions: can the hardware run your workload, how long does it take, and what does it cost. Compute-boundCompute-boundWork whose time is set by how much arithmetic it needs; a chip with more operations per second speeds it up.Where this is taught work favours more FLOP/s; memory-boundMemory bandwidth-boundWork whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.Where this is taught work favours more bandwidth and memory. Answering needs your own workload's mix of prefill and decode.