Mid-level interview questionAsked by candidates · YouTube
What is system-prompt caching, and what does it actually save?
Model answer
The provider keeps the processed prefix, the keys and values prefill computed for your system prompt, and reuses them when the next request starts with the same tokens. You skip that prefill: cheaper input and a faster first token.
Scroll sideways to see the whole diagram
Put what never changes first. The cache can reuse everything up to the first token that differs.
Text version
- Put what never changes first. The cache can reuse everything up to the first token that differs.
- Put something that changes at the top, like a date, and the match ends at token one: nothing is reused.
- This is not a provider quirk. Through attention, each position's keys and values depend on everything before it, so only an identical prefix gives identical values.
Matching rule: Anthropic and OpenAI prompt-caching docs (fetched 2026-10-21). Illustrative prompts.
It only matches an exact prefix, so put static text first and anything that changes, like dates or user data, after it.
Practise the rest
6 Caching questions, 492 in all, each with a model answer. Free.
More Caching interview questions
- Name three caches that reuse work when serving an LLM, and the scope of each.
- Someone added today's date as the first line of every prompt, and the prompt-cache hit rate dropped to zero. Why, and how should prompts be laid out?
- When does a prompt cache make the input bill higher rather than lower?
- A code assistant sends a 2,000-token system prompt and writes long answers. Users complain answers take too long to finish. Will a prefix cache fix it?
- Product wants a semantic answer cache in front of your support bot to cut cost. What risks would you raise, and what would you measure first?