Prompt caching
Also called prompt cache, prefix cache, prefix caching, context cache, context caching
Keeping the KV cache of a prompt's shared opening, such as a system prompt or long document, so later requests that start with the identical prefix skip prefilling it.
Interview questions about Prompt caching
- When does a prompt cache make the input bill higher rather than lower?
When writes aren't followed by reads. Anthropic charges 1.25 times the base input price for a 5-minute cache write and 0.1 times for a read (as of 2026-10-21).
- A code assistant sends a 2,000-token system prompt and writes long answers. Users complain answers take too long to finish. Will a prefix cache fix it?
Mostly not. A prefix cache removes the prefix's prefill, so it cuts TTFT and input cost, not decode.
- What is system-prompt caching, and what does it actually save?
The provider keeps the processed prefix, the keys and values prefill computed for your system prompt, and reuses them when the next request starts with the…
- Name three caches that reuse work when serving an LLM, and the scope of each.
The KV cache works within one request, so each decode step computes only the new token's keys and values: after a 960-token prompt on our laptop CPU, 9.6 ms a…
- Someone added today's date as the first line of every prompt, and the prompt-cache hit rate dropped to zero. Why, and how should prompts be laid out?
A prompt cache reuses only an identical prefix, and the match stops at the first differing token: here, the very first.
- Design a backend where clients submit LLM prompt jobs and poll or receive callbacks, with queueing, model routing, prompt versioning, idempotency and cost control.
Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry.
- Your feature costs too much. Which levers do you compare?
Prompt caching of a shared prefix, routing easy requests to a smaller model, shorter outputs, and fewer input tokens.
- You're under 60 requests a minute but still get 429s. Why?
Limits use a token bucket: capacity refills continuously, so short bursts can be rejected even when the per-minute total is fine.