# KV cache

> KV cache: Stored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.

Canonical: https://llmdojo.dev/glossary/kv-cache

Stored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.

Learn it in the lesson: [The KV cache](https://llmdojo.dev/learn/model-foundations/context/the-kv-cache) (Model Foundations · Context Windows, Prefill and the KV Cache).

## Interview questions about KV cache

- [Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.](https://llmdojo.dev/interview/q/walk-me-through-transformer-attention-causal-and-padding-masks-cross-attention-why)
- [What does the KV cache store, why is using it exact rather than an approximation, and how much does it save?](https://llmdojo.dev/interview/q/what-does-the-kv-cache-store-why-is-using-it-exact-rather)
- [What are multi-head and grouped-query attention, and what does GQA do to the KV cache?](https://llmdojo.dev/interview/q/what-are-multi-head-and-grouped-query-attention-and-what-does-gqa)
- [Here's a small transformer with a few bugs. Fix them, then add a KV cache for decoding.](https://llmdojo.dev/interview/q/here-s-a-small-transformer-with-a-few-bugs-fix-them-then)
- [Implement causal multi-head attention, add a KV cache for token-by-token decoding, then extend it to grouped-query attention.](https://llmdojo.dev/interview/q/implement-causal-multi-head-attention-add-a-kv-cache-for-token-by)
- [Explain FlashAttention, the KV cache and RoPE, and the memory and compute trade-offs each addresses.](https://llmdojo.dev/interview/q/explain-flashattention-the-kv-cache-and-rope-and-the-memory-and-compute)
- [What is attention, exactly how are the queries, keys and values computed, and how does a KV cache speed up decoding?](https://llmdojo.dev/interview/q/what-is-attention-exactly-how-are-the-queries-keys-and-values-computed)
- [Design and implement a KV cache for autoregressive decoding in a Transformer: tensor shapes for prefill and per-step decode, per-layer caches, batching, memory limits and beam search.](https://llmdojo.dev/interview/q/design-and-implement-a-kv-cache-for-autoregressive-decoding-in-a-transformer)

## Related
- [Context window](https://llmdojo.dev/glossary/context-window)
- [Position embedding](https://llmdojo.dev/glossary/position-embedding)
- [Prefill](https://llmdojo.dev/glossary/prefill)
- [Decode](https://llmdojo.dev/glossary/decode)
- [Glossary of every term](https://llmdojo.dev/glossary)