Junior interview questionAsked by candidates · YouTube
Name three caches that reuse work when serving an LLM, and the scope of each.
Model answer
The KV cache works within one request, so each decode step computes only the new token's keys and values: after a 960-token prompt on our laptop CPU, 9.6 ms a step, against 228.8 ms without. A prompt cache keeps a shared opening's keys and values across requests. An answer cache skips the model entirely.
Practise the rest
6 Caching questions, 492 in all, each with a model answer. Free.
More Caching interview questions
- Someone added today's date as the first line of every prompt, and the prompt-cache hit rate dropped to zero. Why, and how should prompts be laid out?
- When does a prompt cache make the input bill higher rather than lower?
- A code assistant sends a 2,000-token system prompt and writes long answers. Users complain answers take too long to finish. Will a prefix cache fix it?
- Product wants a semantic answer cache in front of your support bot to cut cost. What risks would you raise, and what would you measure first?
- What is system-prompt caching, and what does it actually save?