# Benchmarking and the capstone interview questions > 11 benchmarking and the capstone interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/benchmarking Last updated: 2026-10-23 11 questions: 2 junior, 5 mid-level and 4 senior. Taught in [Benchmarking Inference, and the Module Capstone](https://llmdojo.dev/learn/inference-optimization/benchmarks-and-capstone). ## [What does a fair inference benchmark hold fixed?](https://llmdojo.dev/interview/q/what-does-a-fair-inference-benchmark-hold-fixed) Level: Junior **Everything except the thing you changed**: model and precision, prompt and reply lengths, batch, concurrency and arrival rate, warm-up, repeats, environment and unit. Report percentiles, TTFT and TPOT separately. **Diagram: A benchmark you can compare** 1. Decide what is held fixed. Two numbers are comparable only if everything except the thing you are testing is the same. 2. Throw away the first runs. Our scripts ran two or three untimed requests before timing anything. 3. Time the same request many times. One run is one draw from a distribution. 4. Summarise with percentiles. A few slow runs move the mean without telling you how slow they were. 5. Write down everything you didn't vary, so someone else (or you, next month) can tell whether a new number is comparable. After AI Engineering, Ch. 9, "Inference Performance Metrics", and the measuring routine in scripts/generate-latency-data.py. **Lengths alone move results**: on our laptop CPU, the first token took 12.8 ms after 16 prompt tokens and 223.3 ms after 960. ## [An upgrade makes your 960-token prefill 5% faster in one before-and-after run. Do you ship it?](https://llmdojo.dev/interview/q/an-upgrade-makes-your-960-token-prefill-5-faster-in-one-before) Level: Mid-level **Not on that evidence**. Three runs of the same CPU prefill on our laptop, same model, versions, threads and prompt, gave 219.6, 223.3 and 239.4 ms: **9% apart**. Their methods also differed, so no cause could be named. A 5% gap sits **inside that spread**. Interleave several runs of each version with an identical method, and compare the spreads. ## [A vendor says their GPU is twice as fast as your CPU. How would you judge what it buys your workload?](https://llmdojo.dev/interview/q/a-vendor-says-their-gpu-is-twice-as-fast-as-your-cpu) Level: Senior Ask: for **which phase**, prompt length and batch? On our test laptop the Apple GPU's first-token advantage was 1.6× at 16 tokens and 2.3× at 960, but batch-1 decode only about 1.4×, which fits **bandwidth-bound decode** with similar copy ceilings. Short prompts with long replies are **almost all decode**, so the headline barely applies. The book's three questions: can it run the workload, how long does it take, what does it cost. I'd measure my own prefill and decode mix, comparing ratios from one run. ## [A provider's status page shows a p50 TTFT of 180 ms. You need 300 ms for 9 in 10 of your users. What do you do before signing up?](https://llmdojo.dev/interview/q/a-provider-s-status-page-shows-a-p50-ttft-of-180-ms) Level: Senior **Measure it myself**: their number is **their workload on their day**. Latency depends on my prompts and sampling settings, so I'd send my own prompts at my lengths and concurrency and measure p90 TTFT, as the book's criteria table does on an internal prompt dataset. Test at peak concurrency, since some providers throttle. Check quality too: the same Llama models scored differently across providers in a Cerebras experiment. Compare **cost per request**, not per token. ## [In the capstone, continuous batching with at most 8 seats gives a p90 first token of 122 ms against a 300 ms target, yet the planner marks it failing. Why, how do you fix it, and what would you measure first on the real system?](https://llmdojo.dev/interview/q/in-the-capstone-continuous-batching-with-at-most-8-seats-gives-a) Level: Senior It has **no headroom**. Capacity, the highest arrival rate still within 300 ms at p90, is **6 per second** against a target of 9: eight seats fill as traffic grows, and requests queue. A larger batch limit buys capacity until the KV caches break the 2 GB budget. It's simulated from measured steps, but those were measured at a 128-token context while this workload reaches 768, so it's **optimistic**. First real measurement: p90 TTFT and capacity under the real workload. ## [What does perplexity measure, how do tokenizers distort it, and where does it break down as a quality metric?](https://llmdojo.dev/interview/q/what-does-perplexity-measure-how-do-tokenizers-distort-it-and-where-does) Level: Mid-level Perplexity is exp of the mean negative log-likelihood per token, the model's effective branching factor on held-out text. Being **per token**, it isn't comparable across tokenizers: longer tokens mean fewer, harder predictions. Normalise to **bits per byte** to compare. It also measures fit to a text distribution, not helpfulness, correctness or instruction following, so it says little about a chat model's quality, and it shifts with the eval set. ## [Design optimised inference pipelines, and tell me how you would benchmark them.](https://llmdojo.dev/interview/q/design-optimised-inference-pipelines-and-tell-me-how-you-would-benchmark-them) Level: Senior Benchmark the **workload you'll serve**: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt. Report time to first token, inter-token latency and end-to-end at p50 and p99, plus **throughput at a latency SLO**; peak throughput at unbounded latency means little. **Diagram: A benchmark you can compare** 1. Decide what is held fixed. Two numbers are comparable only if everything except the thing you are testing is the same. 2. Throw away the first runs. Our scripts ran two or three untimed requests before timing anything. 3. Time the same request many times. One run is one draw from a distribution. 4. Summarise with percentiles. A few slow runs move the mean without telling you how slow they were. 5. Write down everything you didn't vary, so someone else (or you, next month) can tell whether a new number is comparable. After AI Engineering, Ch. 9, "Inference Performance Metrics", and the measuring routine in scripts/generate-latency-data.py. Change one knob at a time, batch size, precision, tensor-parallel degree, engine, and rerun a quality eval after each, since quantization can quietly cost accuracy. Warm up, pin versions, repeat runs, and profile to explain each result. ## [You're evaluating three agents with different error costs. How do you decide whether precision, recall or F1 is the right metric for each?](https://llmdojo.dev/interview/q/you-re-evaluating-three-agents-with-different-error-costs-how-do-you) Level: Junior Ask which mistake costs more. **Precision** when a false alarm is expensive, like an agent that issues refunds or bans accounts. **Recall** when a miss is expensive, like a safety agent that must catch harmful content. **F1** when both matter and positives are rare, so accuracy misleads. With known costs, weight errors directly. ## [How do you measure agreement between annotators, and when would you use an LLM judge instead of human judges?](https://llmdojo.dev/interview/q/how-do-you-measure-agreement-between-annotators-and-when-would-you-use) Level: Mid-level Agreement tells you whether labels are **reliable** enough to score against. Raw agreement overstates it, since raters agree by chance; **Cohen's kappa** corrects for that with two raters, and Krippendorff's alpha handles many raters and missing labels. Low agreement often means a vague rubric. LLM judges are cheap but biased by position, length and self-preference. Let humans build the gold set; trust a judge once its agreement with them matches human-to-human agreement. ## [Your LLM gives different answers to the same input. How would you make inference reproducible, and how many repeated runs would you need for a stable score?](https://llmdojo.dev/interview/q/your-llm-gives-different-answers-to-the-same-input-how-would-you) Level: Mid-level First remove the randomness you control: **greedy decoding** or a fixed seed, and a pinned model, prompt and parameters. Even then, GPU serving isn't bitwise deterministic, because batch composition changes floating-point reduction order, so cache outputs for exact replay. For the score, treat runs as samples. Measure the **standard deviation** across repeats in a pilot, then pick n so the standard error, sd over root n, is well below the difference you need to detect. ## [Design simulation experiments to evaluate a self-driving agent's safety. Which scene parameters do you vary, how do you sample them, and how do you estimate a rare collision rate without bias?](https://llmdojo.dev/interview/q/design-simulation-experiments-to-evaluate-a-self-driving-agent-s-safety-which) Level: Mid-level Vary what drives risk: actor types and behaviours, speeds and gaps, occlusions, road geometry, weather and sensor noise. Sample from **real-world distributions** fitted to driving logs, so results mean something. Collisions are rare, so plain Monte Carlo wastes runs; use **importance sampling** to oversample dangerous regions, then weight each run by its likelihood ratio, keeping the collision-rate estimate **unbiased**, with confidence intervals. Validate the simulator against logged events. ## Related - [All interview questions](https://llmdojo.dev/interview)