Estimate the KV cache bytes per token for Llama-3.1-8B, and how many concurrent users one H100 80 GB can hold at a 4k-token average context.
- 1 min — write the formula and the model's shape (layers, KV heads, head_dim, dtype)
- 2 min — compute bytes/token and usable memory after weights and headroom
- 2 min — divide by average live context; show the FP8 KV and long-context variants
- 2 min — policy: context caps, separate pools, the metrics that would prove you wrong
KV bytes per token = 2 (K and V) × layers × KV heads × head_dim × bytes per element. Llama-3.1-8B has 32 layers, 8 KV heads (grouped-query attention) and head_dim 128, so in BF16 that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token. Concurrency is then (GPU memory × utilisation − weights − activation headroom) ÷ (bytes per token × average live context).
Capacity planning for an internal code-search assistant on one H100: you need to know whether one card carries the expected 80 concurrent sessions before you buy a second. The KV budget, not FLOPs, decides it — decode is memory-bound and every live token pins 128 KiB until the request finishes.
The answer that separates levels is what you do with the number. Set max_model_len to the p99 real context, not 128k, or one long-context user evicts dozens of chat users. Turn on FP8 KV (halves bytes per token) and prefix caching (shared system prompts are stored once under paged KV). Organisationally: the serving platform team owns the per-tier context caps and prices long context as the multiple of a chat session it really is; product teams request a tier, they don't pick max_model_len.
Assumptions: Llama-3.1-8B, BF16 weights (8.0B params × 2 B ≈ 16 GB), one H100 80 GB, vLLM with gpu_memory_utilization 0.90, ~4 GB activations/CUDA graphs, average live context 4,096 tokens (prompt + generated so far).
- Usable memory: 80 × 0.90 = 72 GB → minus 16 GB weights → minus 4 GB → ~52 GB for KV.
- Bytes per token: 2 × 32 × 8 × 128 × 2 B = 131,072 B (128 KiB).
- Live tokens: 52 × 10⁹ ÷ 131,072 ≈ 397k tokens.
- Concurrency: 397k ÷ 4,096 ≈ ~97 sessions in BF16 KV; ~190 with FP8 KV (64 KiB/token).
- Same card at 32k context: ~12 sessions (BF16) — one 128k user costs as much KV as 32 chat users.
- Cross-check for 70B (80 layers, 8 KV heads): 2 × 80 × 8 × 128 × 2 = 320 KiB/token, 2.5× the 8B.
What the answer sounds like at each level
Gets the formula right and computes 128 KiB/token and roughly 100 concurrent sessions, remembering to subtract the weights.
Adds activation and fragmentation headroom, shows FP8 KV doubling concurrency, and notes that average context includes generated tokens and grows during a session.
Turns it into policy: per-tier context caps, a separate long-context pool, pricing by KV-seconds, and the preemption/KV-usage alerts that tell you the estimate has gone stale.
Follow-ups the interviewer will push
1Now the product wants a 128k-context tier on the same fleet.
128k × 128 KiB = 16 GiB per session, so a card holds 3 such users with BF16 KV. Don't raise max_model_len globally — run a separate long-context pool with FP8 KV, admit by quota, and price it at ~30× a chat session.
2Now it's Llama-3.1-70B on 2×H100 with tensor parallelism.
320 KiB/token in BF16, 160 KiB in FP8. 160 GB × 0.9 − 70 GB FP8 weights − ~8 GB ≈ 66 GB → ~410k tokens at FP8 KV → ~100 sessions at 4k. Tensor parallelism splits KV heads across GPUs, so per-GPU bytes halve but the total is what matters.
3Observed concurrency is 40% lower than your estimate. Why?
Usually the average context was measured on prompts only — generated tokens and multi-turn history count too — or max_model_len reservations and block fragmentation. Check the KV-usage gauge at p95 and the preemption counter before buying hardware.
Red flags interviewers listen for
- Uses 32 query heads instead of 8 KV heads, overstating the cache 4× — misses that GQA exists for exactly this reason.
- Forgets to subtract model weights, or assumes 100% of HBM is usable.
- Treats concurrency as a FLOPs question rather than a memory question.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Arithmetic | 128 KiB/token and ~100 sessions within 20%, with units shown at every step. |
| Assumptions | States dtype, utilisation, activation headroom and that context includes generated tokens. |
| Levers | Names FP8 KV, prefix caching, per-tier max_model_len, and quantifies each. |
| Operational close | Gives KV-usage >85% at p95 and preemption >1% as the alerts that trigger more capacity. |
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023paper
- The Llama 3 Herd of Models — Llama Team, Meta, 2024paper