Caching repeated work interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 caching repeated work questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Caching Repeated Work: Prompts, Prefixes and Answers teaches the ideas behind them.Updated
Model answer
The KV cache works within one request, so each decode step computes only the new token's keys and values: after a 960-token prompt on our laptop CPU, 9.6 ms a step, against 228.8 ms without. A prompt cache keeps a shared opening's keys and values across requests. An answer cache skips the model entirely.
Model answer
A prompt cache reuses only an identical prefix, and the match stops at the first differing token: here, the very first. It isn't a provider quirk: a token's keys and values depend on every token before it.
Scroll sideways to see the whole diagram
Put what never changes first. The cache can reuse everything up to the first token that differs.
Text version
- Put what never changes first. The cache can reuse everything up to the first token that differs.
- Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
- This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.
Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.
Order prompts static first: fixed instructions and tools, then shared reference material, then per-request parts like the date, retrieved snippets and the question.
Model answer
When writes aren't followed by reads. Anthropic charges 1.25 times the base input price for a 5-minute cache write and 0.1 times for a read (as of 2026-10-21). A 2,000-token prefix sent once costs 24% more than not caching. Rare traffic is the same: the entry expires, so every request pays to write it again. Shared by 10 requests, a 10,000-token prefix saves 77%.
Model answer
Mostly not. A prefix cache removes the prefix's prefill, so it cuts TTFT and input cost, not decode. In our CPU estimate with 128 shared and 64 unique tokens and 400 output tokens, the first token came about 48% sooner but the reply finished only about 1% sooner: 3.32 s to 3.30 s. That estimate is optimistic, too: it ignores attention to the cached prefix and loading the cache.
To finish sooner, change decode: shorter outputs, faster hardware, or fewer bytes per weight.
Model answer
An answer cache serves a reply written for an earlier request. Risks: stale answers to time-sensitive queries; leaks, like the book's return-policy answer built from user X's details and served to user Y; and, with semantic matching, wrong hits. Embeddings capture topic more than truth: with all-MiniLM-L6-v2, "The Earth orbits the Sun." and its reverse scored 0.982, while the best true paraphrase scored 0.645.
I'd measure hit rate and wrong-hit rate on our own queries, and keep personal queries out.
Model answer
The provider keeps the processed prefix, the keys and values prefill computed for your system prompt, and reuses them when the next request starts with the same tokens. You skip that prefill: cheaper input and a faster first token.
Scroll sideways to see the whole diagram
Put what never changes first. The cache can reuse everything up to the first token that differs.
Text version
- Put what never changes first. The cache can reuse everything up to the first token that differs.
- Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
- This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.
Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.
It only matches an exact prefix, so put static text first and anything that changes, like dates or user data, after it.