KV cache
Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.
When an Large Language Model (LLM) writes a reply one Token at a time, each new token needs to look back at everything before it. Recomputing all of that each time would be very slow. The KV cache saves the intermediate results (called keys and values) for earlier tokens so they can be reused.
It makes generation fast, but it uses a lot of GPU memory, which grows with the length of the conversation and the number of users at once.
Keeping running notes during a long meeting. Without notes, every time you speak you would replay the whole recording; with notes, you glance at them — but the notebook gets thicker with every minute and every attendee.
Thinking the KV cache stores conversation memory across sessions or makes the model 'remember'. It is a per-request computation cache; reuse across requests only happens via exact-prefix caching.
Bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes. Llama-3.1-70B (80 layers, 8 KV heads, head_dim 128, BF16) = 320 KB/token, so one 128k context holds ~42 GB — more than a quarter of the BF16 weights. Decode is memory-bandwidth-bound: every step reads the weights plus the whole cache of every sequence in the batch, so long contexts slow every token, not just the first.
Mitigations, in the order to reach for them: grouped-query attention (8 KV heads instead of 64 is 8× smaller), FP8 KV cache (2×), PagedAttention (vLLM's 16-token blocks eliminate fragmentation, which the paper measured wasting 60–80% of KV memory in earlier systems), prefix caching (share KV blocks across requests with identical prefixes — vLLM automatic prefix caching, SGLang RadixAttention), latent attention (DeepSeek's MLA compresses KV further), and per-tier context caps. When the cache fills, the scheduler preempts sequences (recompute or swap), which shows up as tail-latency spikes. Provider Prompt caching is the same mechanism across API calls, which is why it requires byte-identical prefixes.
vLLM or SGLang, paged KV, FP8 KV cache, prefix caching on, max_model_len set to the p99 real context (not the model maximum), gpu_memory_utilization 0.9, and a separate pool or tier for contexts above 32k.
2×H100 (160 GB), Llama-3.1-70B in FP8 (~70 GB weights) with FP8 KV (160 KB/token). With gpu_memory_utilization 0.9 and ~5 GB overhead, ~70 GB is left for KV ≈ 430k live tokens: ~105 concurrent users at 4k, ~13 at 32k, ~3 at 128k. Serving 16 users at 4k needs 11 GB of KV; 16 at 32k needs 86 GB (over budget); 16 at 128k needs 344 GB — more than two whole GPUs.
Do the per-token arithmetic out loud, then turn it into concurrency per GPU and price. Staff line: 'Weights are a fixed cost; KV is a per-user, per-token cost, and at long context it is what fills the GPU — so I price and quota long-context users separately.' Expect: 'how does prompt caching relate?' — same KV reuse across requests, exact-prefix only.
| Context | KV per sequence (BF16) | KV per sequence (FP8) | Users in 70 GB (FP8) |
|---|---|---|---|
| 4k | 1.3 GB | 0.67 GB | ~105 |
| 32k | 10.7 GB | 5.4 GB | ~13 |
| 128k | 42.9 GB | 21.5 GB | ~3 |
Numbers worth knowing
- Llama-3.1-70B KV
- 320 KB/token BF16, 160 KB FP880 layers × 8 KV heads × 128 dim
- 128k context, one sequence
- ~42 GB BF16Llama-3.1-70B
- Fragmentation waste before paging
- 60–80% of KV memorymeasured in the PagedAttention paper
- GQA saving
- 8×8 KV heads vs 64 query heads (70B)
Try the numbers
Defaults: a 70B GQA model (80 layers, 8 KV heads, head dim 128) in BF16, with ~85 GB left after FP8 weights on 2×H100. Ignores activation and fragmentation overhead (paged attention keeps the latter near zero).
How to decide
| When | Do this |
|---|---|
| KV usage > 85% at p95 | FP8 KV, then cap max_model_len per tier, then add GPUs. |
| Many requests share a long system prompt or documents | Turn on prefix caching and put the shared part first, byte-identical. |
| A long-context tier launches | Give it its own pool and quota; never raise max_model_len globally. |
Cost maths
2×H100 (160 GB). 70B FP8 weights ≈ 70 GB → ~85 GB left for KV at FP8 (160 KB/token) ≈ 530k live tokens before the 0.9 utilisation margin: ~130 concurrent users at 4k context, or 4 at 128k. At an assumed $6/hour for the pair, a 4k user costs ~$0.05/hour of occupancy and a 128k user ~$1.50/hour — ~32×. Price and quota them that way.
Operating it
- KV usage > 85% at p95
- Preemption rate > 1%
- Inter-token latency p95 doubles as context grows
- GPU KV cache usage > 85% for 5 minutes
- Preempted sequences > 1% of requests
- ITL p95 by context-length bucket — alert on 2× baseline
- Prefix-cache hit rate < 50% on routes designed for it
When it goes wrong
p99 latency rose 6× the week a 200k-context tier launched; short-chat users complained, not the long-context ones.
vLLM preemption counter and the KV-usage gauge, both pinned during business hours.
max_model_len was raised globally; three long sessions held enough KV to evict forty short ones, which were preempted and recomputed repeatedly.
Per-tier context caps, FP8 KV, and a separate long-context pool with its own quota and price.
Related terms
Where it shows up
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023paper
- SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng et al., 2023paper
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA) — DeepSeek-AI, 2024paper
- The Llama 3 Herd of Models — Llama Team, Meta, 2024paper
Check yourself
Three questions. Every option is explained once you answer.