Topic 2

Measuring decode and prefill on a roofline

To judge a measurement, you need the ceiling it is up against. We measured two on each device: a large memory copy (the most bytes per second it reached) and a large matrix multiply (the most operations per second). Apple publishes 153 GB/s of memory bandwidth for the M5; our best copy reached 122.3 GB/s on the CPU and 129.6 GB/s on the GPU.

A rooflineRooflineA chart of achievable operations per second against arithmetic intensity: a sloped roof set by memory bandwidth and a flat roof set by peak compute, meeting at the ridge.Where this is taught draws both ceilings on one chart. Work with few operations per byteArithmetic intensityOperations done per byte read from memory. Low intensity means bandwidth is the limit; high intensity means compute is.Where this is taught sits under the sloped roof (bandwidth); work with many sits under the flat roof (compute). The roofs meet at the ridge: 16 operations per byte on the CPU, 27 on the GPU.

Decode is on the bandwidth side. At batch 1, a step took 9.1 ms on the CPU: 507 MB in 9.1 ms is 55.7 GB/s, 46% of the measured copy ceiling. The GPU did 6.3 ms, 80.5 GB/s, 62%. Both sit at about 0.5 operations per byte, far left of either ridge. We did not break down where the rest of the time goes.

PrefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught is on the compute side. A 960-token prompt took 239.4 ms on the CPU: about 998 billion operations per second, 51% of the matrix-multiply ceiling, at about 480 operations per byte. That is 0.25 ms per prompt tokenTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught against 9.1 ms per decode stepDecodeThe sequential phase of generation: one forward pass per new token.Where this is taught.

AI Engineering calls the bandwidth share MBUModel bandwidth utilization (MBU)The share of peak memory bandwidth inference uses: parameters × bytes per parameter × tokens per second, over the peak, for one sequence.Where this is taught (model bandwidth utilizationUtilizationThe share of a resource's capacity actually in use. A rented machine that sits idle still bills by the hour, so low utilization raises the cost per request.Where this is taught): parameters × bytes per parameter × tokens/s, over the peak bandwidth. Its example: a 7B model in 16-bit at 100 tokens/s uses 700 GB/s, 70% of an A100-80GB's 2 TB/s. The formula assumes one sequence; with a batch, the weights are read once for all of them.