Benchmarking and the capstone interview questions

Q: Your LLM gives different answers to the same input. How would you make inference reproducible, and how many repeated runs would you need for a stable score?

First remove the randomness you control: greedy decoding or a fixed seed, and a pinned model, prompt and parameters. Even then, GPU serving isn't bitwise deterministic, because batch composition changes floating-point reduction order, so cache outputs for exact replay. For the score, treat runs as samples. Measure the standard deviation across repeats in a pilot, then pick n so the standard error, sd over root n, is well below the difference you need to detect.

Question 1

What does a fair inference benchmark hold fixed?

Accepted Answer

Everything except the thing you changed: model and precision, prompt and reply lengths, batch, concurrency and arrival rate, warm-up, repeats, environment and unit. Report percentiles, TTFT and TPOT separately. Lengths alone move results: on our laptop CPU, the first token took 12.8 ms after 16 prompt tokens and 223.3 ms after 960.

Question 2

An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?

Accepted Answer

Not on that evidence. Three runs of the same CPU prefill on our laptop, same model, versions, threads and prompt, gave 219.6, 223.3 and 239.4 ms: 9% apart. Their methods also differed, so no cause could be named. A 5% gap sits inside that spread. Interleave several runs of each version with an identical method, and compare the spreads.

Question 3

A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?

Accepted Answer

Ask: for which phase, prompt length and batch? On our test laptop the Apple GPU's first-token advantage was 1.6× at 16 tokens and 2.3× at 960, but batch-1 decode only about 1.4×, which fits bandwidth-bound decode with similar copy ceilings. Short prompts with long replies are almost all decode, so the headline barely applies. The book's three questions: can it run the workload, how long does it take, what does it cost. I'd measure my own prefill and decode mix, comparing ratios from one run.

Question 4

A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?

Accepted Answer

Measure it myself: their number is their workload on their day. Latency depends on my prompts and sampling settings, so I'd send my own prompts at my lengths and concurrency and measure p90 TTFT, as the book's criteria table does on an internal prompt dataset. Test at peak concurrency, since some providers throttle. Check quality too: the same Llama models scored differently across providers in a Cerebras experiment. Compare cost per request, not per token.

Question 5

In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?

Accepted Answer

It has no headroom. Capacity, the highest arrival rate still within 300 ms at p90, is 6 per second against a target of 9: eight seats fill as traffic grows, and requests queue. A larger batch limit buys capacity until the KV caches break the 2 GB budget. It's simulated from measured steps, but those were measured at a 128-token context while this workload reaches 768, so it's optimistic. First real measurement: p90 TTFT and capacity under the real workload.

Question 6

What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?

Accepted Answer

Perplexity is exp of the mean negative log-likelihood per token, the model's effective branching factor on held-out text. Being per token, it isn't comparable across tokenizers: longer tokens mean fewer, harder predictions. Normalise to bits per byte to compare. It also measures fit to a text distribution, not helpfulness, correctness or instruction following, so it says little about a chat model's quality, and it shifts with the eval set.

Question 7

Design optimised inference pipelines, and tell me how you would benchmark them.

Accepted Answer

Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt. Report time to first token, inter-token latency and end-to-end at p50 and p99, plus throughput at a latency SLO; peak throughput at unbounded latency means little. Change one knob at a time, batch size, precision, tensor-parallel degree, engine, and rerun a quality eval after each, since quantization can quietly cost accuracy. Warm up, pin versions, repeat runs, and profile to explain each result.

Question 8

You're evaluating three agents with different error costs. How do you decide whether precision, recall or F1 is the right metric for each?

Accepted Answer

Ask which mistake costs more. Precision when a false alarm is expensive, like an agent that issues refunds or bans accounts. Recall when a miss is expensive, like a safety agent that must catch harmful content. F1 when both matter and positives are rare, so accuracy misleads. With known costs, weight errors directly.

Question 9

How do you measure agreement between annotators, and when would you use an LLM judge instead of human judges?

Accepted Answer

Agreement tells you whether labels are reliable enough to score against. Raw agreement overstates it, since raters agree by chance; Cohen's kappa corrects for that with two raters, and Krippendorff's alpha handles many raters and missing labels. Low agreement often means a vague rubric. LLM judges are cheap but biased by position, length and self-preference. Let humans build the gold set; trust a judge once its agreement with them matches human-to-human agreement.

Question 10

Your LLM gives different answers to the same input. How would you make inference reproducible, and how many repeated runs would you need for a stable score?

Accepted Answer

First remove the randomness you control: greedy decoding or a fixed seed, and a pinned model, prompt and parameters. Even then, GPU serving isn't bitwise deterministic, because batch composition changes floating-point reduction order, so cache outputs for exact replay.

For the score, treat runs as samples. Measure the standard deviation across repeats in a pilot, then pick n so the standard error, sd over root n, is well below the difference you need to detect.

Question 11

Design simulation experiments to evaluate a self-driving agent's safety. Which scene parameters do you vary, how do you sample them, and how do you estimate a rare collision rate without bias?

Accepted Answer

Vary what drives risk: actor types and behaviours, speeds and gaps, occlusions, road geometry, weather and sensor noise. Sample from real-world distributions fitted to driving logs, so results mean something. Collisions are rare, so plain Monte Carlo wastes runs; use importance sampling to oversample dangerous regions, then weight each run by its likelihood ratio, keeping the collision-rate estimate unbiased, with confidence intervals. Validate the simulator against logged events.