Senior interview question
A code assistant sends a 2,000-token system prompt and writes long answers. Users complain answers take too long to finish. Will a prefix cache fix it?
Model answer
Mostly not. A prefix cache removes the prefix's prefill, so it cuts TTFT and input cost, not decode. In our CPU estimate with 128 shared and 64 unique tokens and 400 output tokens, the first token came about 48% sooner but the reply finished only about 1% sooner: 3.32 s to 3.30 s. That estimate is optimistic, too: it ignores attention to the cached prefix and loading the cache.
To finish sooner, change decode: shorter outputs, faster hardware, or fewer bytes per weight.
Practise the rest
6 Caching questions, 492 in all, each with a model answer. Free.
More Caching interview questions
- Name three caches that reuse work when serving an LLM, and the scope of each.
- Someone added today's date as the first line of every prompt, and the prompt-cache hit rate dropped to zero. Why, and how should prompts be laid out?
- When does a prompt cache make the input bill higher rather than lower?
- Product wants a semantic answer cache in front of your support bot to cut cost. What risks would you raise, and what would you measure first?
- What is system-prompt caching, and what does it actually save?