Topic 7

Capstone: writing up the decision, and wrap-up

A configuration isn't finished until someone else can check it. Write it up in your notes file, in this order:

• Decision. Device, batching policy, batch limit.
• Evidence. The four checks, with the number for each and its margin from the target.
• Assumptions. At least three from the planner's list that your decision depends on, and one of your own about the real service.
• Trade-offs. The passing configurations you rejected, and why.
• What would change it. The first measurement you would run on the real system, and what result would overturn the decision.

What "done" looks like

• The decision is one configuration that meets all four targets in the planner, with every number copied from the screen.
• Task 3 names both edges of the passing range and the target that sets each one.
• Task 4 says which check the CPU never passes, with its best value.
• The assumptions include that decode stepsDecodeThe sequential phase of generation: one forward pass per new token.Where this is taught were measured at a 128-token context while this workload reaches 768 tokensTokenA piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.Where this is taught, and say which way that biases the result.
• The trade-off says what the extra seats cost (memory) and buy (capacity).

Summary

• Measure what users wait for (TTFTTime to first token (TTFT)How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.Where this is taught, TPOTTime per output token (TPOT)The average time between output tokens after the first one: how fast a streamed reply flows.Where this is taught) as percentilesPercentile (p50, p99)pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.Where this is taught, and count goodputGoodputThe rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.Where this is taught against an SLOService level objective (SLO)A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.Where this is taught, not raw throughputThroughputWork done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.Where this is taught.
• Explain the time: decode is bandwidth-boundMemory bandwidth-boundWork whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.Where this is taught, prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught compute-boundCompute-boundWork whose time is set by how much arithmetic it needs; a chip with more operations per second speeds it up.Where this is taught; batching shares the weight read until KV-cache memory caps it. Precision, caching and cost per requestCost per requestWhat one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.Where this is taught each change the bill.
• Benchmark with the workload fixed, repeated and recorded, and decide against stated targets with the assumptions written down.

Next

Module 03, Prompt and Context Engineering, is planned.