Attention
The transformer mechanism that lets each token weigh every earlier token in the context when building its representation.
Attention is how a language model decides which earlier words matter for the word it is working on. In 'The lease renews unless it ends', attention lets 'it' connect strongly to 'lease', so the model knows what is being talked about.
Every Token looks back at the others and takes a weighted mix of their information, with bigger weights on the more relevant ones.
Reading with a highlighter: for each word, you highlight the earlier words that help you understand it.
Reading attention weights as the reason for an answer. They show information flow in one head of one layer, not a faithful explanation.
Each token is projected into query (Q), key (K) and value (V) vectors. Scores are QKᵀ / √d_k, a causal mask blanks future positions, softmax turns each row into weights summing to 1, and the weights mix the V vectors. Multi-head attention runs this in parallel with different projections.
Compute is quadratic in sequence length (an n × n score matrix per head per layer), but at serving time memory dominates: the KV cache stores K and V for every past token, 2 × layers × kv_heads × head_dim × bytes per token. Grouped-query attention (GQA) shares K/V across query heads: Llama-3.1-70B has 64 query heads but 8 KV heads, an 8× smaller cache. FlashAttention computes exact attention in tiles without materialising the n × n matrix, cutting memory traffic. RoPE and its scaling variants decide how well attention generalises to long contexts. The same mechanism, without the causal mask, powers encoders and cross-encoders.
Serve GQA models on vLLM or SGLang with FlashAttention kernels, paged KV, FP8 KV cache where quality holds on your evals, prefix caching on, and max_model_len set to the p99 real context rather than the model maximum.
Llama-3.1-8B: 32 layers × 8 KV heads × 128 dims × 2 (K and V) × 2 bytes = 128 KB per token. A 32k-token code-review prompt needs 4 GB of KV; 32 concurrent users at 8k each need 32 GB, twice the 16 GB of weights. That memory, not FLOPs, caps how many long requests one 80 GB GPU can serve.
Staff rounds rarely ask for the derivation; they test the consequences. Commit to: long context is a KV-memory problem at serving time, GQA and FP8 KV are the first levers, prefix caching works because K/V for a shared prefix are identical, and position affects recall. Expect 'how many concurrent 32k users fit on one H100 with an 8B model?': work it out from bytes per token.
| The | lease | renews | unless | it | ends | |
|---|---|---|---|---|---|---|
| The | 1.00 | · | · | · | · | · |
| lease | 0.35 | 0.65 | · | · | · | · |
| renews | 0.10 | 0.60 | 0.30 | · | · | · |
| unless | 0.05 | 0.25 | 0.45 | 0.25 | · | · |
| it | 0.04 | 0.70 | 0.10 | 0.06 | 0.10 | · |
| ends | 0.03 | 0.30 | 0.20 | 0.27 | 0.12 | 0.08 |
Numbers worth knowing
- KV bytes per token
- 2 × layers × kv_heads × head_dim × bytesLlama-3.1-8B BF16 = 128 KB; Llama-3.1-70B BF16 = 320 KB
- GQA saving
- 8×Llama-3.1-70B: 64 query heads share 8 KV heads
- Attention compute
- O(n²) per layerdoubling the prompt quadruples score-matrix work in prefill
How to decide
| When | Do this |
|---|---|
| Serving long contexts to many users | Pick a GQA model, enable FP8 KV and paged attention; cap per-tier context. |
| Many requests share a long prefix | Enable prefix caching so K/V for the prefix are computed once. |
| Need to explain why the model answered something | Use citations and ablations, not attention maps. |
Cost maths
Concurrency = free KV memory ÷ (bytes per token × context). One H100 80 GB, Llama-3.1-8B BF16: 16 GB weights, ~56 GB usable for KV at 0.9 utilisation ≈ 440k tokens ≈ 53 users at 8k context, or 13 at 32k. At an assumed $2.50/GPU-hour (illustrative), that is ~$0.05 per 8k-user-hour versus ~$0.19 per 32k-user-hour: price long-context tiers accordingly.
Operating it
- KV usage above 85% at p95
- Scheduler preemptions above 1% of requests
- Inter-token latency p95 doubles as average context grows
- KV cache usage p95: alert above 85%
- Preemptions per minute: alert above 1% of requests
- Inter-token latency p95 by context bucket: alert on 2× baseline
When it goes wrong
Throughput on a code-search assistant fell 70% the day 'whole repository' mode launched; short requests timed out.
vLLM KV-usage gauge pinned at 100% and the preemption counter climbing.
max_model_len was raised from 16k to 128k for everyone; a handful of 100k-token sessions held most of the KV cache and evicted dozens of short ones.
Separate long-context pool with its own quota, FP8 KV, and per-tier context caps at the gateway.
Related terms
Where it shows up
Sources
- Attention Is All You Need — Vaswani et al., 2017paper
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Dao et al., 2022paper
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.