Mid-level interview question
When is caching whole answers dangerous?
Model answer
When the answer depends on who asked. The book's example: a return-policy answer built from user X's membership, cached as generic, then served to user Y.
Scroll sideways to see the whole diagram
AI Engineering: exact caching suits answers that are the same for everyone, and is especially appealing for multi-step or slow work like retrieval, SQL or web search.
Text version
- AI Engineering: exact caching suits answers that are the same for everyone, and is especially appealing for multi-step or slow work like retrieval, SQL or web search.
- User-specific queries (“What's the status of my recent order?”) are unlikely to be reused, and time-sensitive ones go stale. Many teams train a classifier to decide what to cache.
- The book's warning: a return-policy question looks generic, but the answer used user X's membership details. Cached as generic, it reveals X's information to the next user who asks.
- So a cache key must include everything the answer depended on, or the answer must not be cached.
Key the cache on everything the answer depended on, skip user-specific and time-sensitive queries, and invalidate on prompt or data changes.
Practise the rest
6 Budgets questions, 492 in all, each with a model answer. Free.