Senior interview question
Product wants a semantic answer cache in front of your support bot to cut cost. What risks would you raise, and what would you measure first?
Model answer
An answer cache serves a reply written for an earlier request. Risks: stale answers to time-sensitive queries; leaks, like the book's return-policy answer built from user X's details and served to user Y; and, with semantic matching, wrong hits. Embeddings capture topic more than truth: with all-MiniLM-L6-v2, "The Earth orbits the Sun." and its reverse scored 0.982, while the best true paraphrase scored 0.645.
I'd measure hit rate and wrong-hit rate on our own queries, and keep personal queries out.
Practise the rest
6 Caching questions, 492 in all, each with a model answer. Free.
More Caching interview questions
- Name three caches that reuse work when serving an LLM, and the scope of each.
- Someone added today's date as the first line of every prompt, and the prompt-cache hit rate dropped to zero. Why, and how should prompts be laid out?
- When does a prompt cache make the input bill higher rather than lower?
- A code assistant sends a 2,000-token system prompt and writes long answers. Users complain answers take too long to finish. Will a prefix cache fix it?
- What is system-prompt caching, and what does it actually save?