Lesson 4 · 8 topics · 20 min · intermediate

Caching Repeated Work: Prompts, Prefixes and Answers

You will be able to

Tell apart the three scopes of reuse: the KV cache within a request, a prompt (prefix) cache across requests, and an answer cache in front of the model.
Explain why a prompt cache needs an identical prefix, and order a prompt static-first so the most tokens are reused.
Estimate how much a cached prefix shortens time to first token from measured prefill curves, and say why the estimate is optimistic.
Compute the input bill with and without a prompt cache from write and read price multipliers, and say when caching costs more.
Describe exact and semantic caching of whole answers and their risks: stale answers, leaks between users, and wrong semantic matches.
Recognise workloads where caching doesn't help: no repeated prefix, a prefix below the minimum, expired entries, or decode-dominated replies.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization", section "Inference Service Optimization", subsection "Prompt caching"; Ch. 10 "AI Engineering Architecture and User Feedback", "Step 4. Reduce Latency with Caches": "Exact caching" and "Semantic caching".
Prompt caching · Anthropic
Anthropic. Claude Platform documentation. Used: Fetched 2026-10-21: identical-prefix rule, cache order, pricing multipliers, lifetime, minimum length, organization and workspace isolation. Open source
Prompt caching · OpenAI
OpenAI. OpenAI API documentation. Used: Fetched 2026-10-21: prefix-match rule, static-first advice, cached-input discount, minimum length, lifetime, no sharing across organizations. Open source

You will be able to

Tell apart the three scopes of reuse: the KV cache within a request, a prompt (prefix) cache across requests, and an answer cache in front of the model.

Explain why a prompt cache needs an identical prefix, and order a prompt static-first so the most tokens are reused.

Estimate how much a cached prefix shortens time to first token from measured prefill curves, and say why the estimate is optimistic.

Compute the input bill with and without a prompt cache from write and read price multipliers, and say when caching costs more.

Describe exact and semantic caching of whole answers and their risks: stale answers, leaks between users, and wrong semantic matches.

Recognise workloads where caching doesn't help: no repeated prefix, a prefix below the minimum, expired entries, or decode-dominated replies.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 9 "Inference Optimization", section "Inference Service Optimization", subsection "Prompt caching"; Ch. 10 "AI Engineering Architecture and User Feedback", "Step 4. Reduce Latency with Caches": "Exact caching" and "Semantic caching".

Prompt caching · Anthropic

Anthropic. Claude Platform documentation. Used: Fetched 2026-10-21: identical-prefix rule, cache order, pricing multipliers, lifetime, minimum length, organization and workspace isolation. Open source

Prompt caching · OpenAI

OpenAI. OpenAI API documentation. Used: Fetched 2026-10-21: prefix-match rule, static-first advice, cached-input discount, minimum length, lifetime, no sharing across organizations. Open source