Walk through inference latency: TTFT, throughput, batching and the KV cache.
- 2 min — prefill vs decode and the latency formula
- 3 min — the five serving techniques
- 1 min — KV memory math
- 1 min — SLOs and ownership
Prefill processes the prompt in parallel and emits the first token — compute-bound, sets TTFT. Decode emits one token per step, reading weights plus the KV cache each step — memory-bandwidth-bound, sets inter-token latency (ITL).
latency ≈ TTFT + output_tokens × ITL
Modern serving (vLLM, SGLang, TensorRT-LLM) stacks five techniques: continuous batching (requests join and leave the batch every step, not per batch); PagedAttention (KV stored in fixed-size blocks like virtual memory, cutting fragmentation waste from 60–80% to under 4%); prefix caching (reuse KV blocks for a shared system prompt or document across requests); chunked prefill (split long prompts into chunks interleaved with decode steps so one 100k prompt doesn't stall every streaming user); prefill/decode disaggregation (separate GPU pools for compute-bound prefill and bandwidth-bound decode, KV transferred between them, so each pool is sized to its own SLO).
A support chat feels slow though tokens stream fast. Gateway traces: TTFT p95 2.8 s, ITL p95 35 ms. Cause: a 6k-token system prompt plus 4k retrieved context re-prefilled every turn. Fixes: provider prompt caching on the stable 6k prefix (hosted) or prefix caching (self-hosted), trim retrieval from 12 to 6 chunks. TTFT p95 → 0.9 s with no model change. On the self-hosted path, enabling chunked prefill also stopped a nightly batch of 80k-token summaries from spiking chat ITL p99 from 40 ms to 900 ms.
- Track TTFT, ITL and end-to-end at p50/p95/p99 separately — streaming UX lives on TTFT (target <1 s), reading-speed UX on ITL (<50 ms ≈ 20 tok/s), batch jobs on throughput.
- KV memory, not weights, caps concurrency on long contexts: Llama-3.1-70B holds 320 KB/token in BF16, 160 KB in FP8.
- Throughput vs latency is the core trade: larger batches cut $/token and raise ITL. Pick batch limits from the SLO, not the benchmark.
- Org scope: one inference platform team owns serving config (engine version, KV dtype, max batch, chunk size) for all product teams; buy hosted for spiky or frontier workloads, build only for steady high-volume open-weights traffic. Roll engine upgrades through canary pools with latency SLO guards.
- Where it breaks: load tests with short prompts. Real long-context traffic has different TTFT, KV pressure and preemption behaviour.
Llama-3.1-70B FP8 on 2×H100 (160 GB): weights ≈ 70 GB, ~80 GB usable KV at FP8 160 KB/token ≈ 500k live tokens. Chat users at 4k context → ~125 concurrent sequences. With a 6k shared prefix cached once, each user's unique KV is ~2k tokens → ~250 concurrent. Decode at batch 125: per step reads ~70 GB weights + ~80 GB KV ≈ 150 GB over 6.7 TB/s ≈ 22 ms → ~5,600 output tok/s aggregate (upper bound).
What the answer sounds like at each level
Separates TTFT from per-token latency and knows batching raises throughput.
Explains prefill vs decode boundness, KV memory limits, continuous batching, PagedAttention and prefix caching with numbers.
Designs serving with chunked prefill and disaggregation against explicit SLOs, owns it as a shared platform, and load-tests on production-shaped traffic.
Follow-ups the interviewer will push
1When is prefill/decode disaggregation worth it?
When long prompts and strict ITL SLOs coexist at scale: prefill bursts no longer interfere with decode. Below a few dozen GPUs, chunked prefill on a unified pool is simpler and close enough.
2Now 10× concurrent users on the same GPUs. What breaks first?
KV capacity: the scheduler preempts or queues, TTFT p99 explodes. Mitigate with FP8 KV, prefix caching, max_model_len at p99 real context, and then more replicas.
3Hosted API is slow at TTFT. What can you do without self-hosting?
Cache the stable prefix, shrink and reorder context so the stable part comes first, stream, and use a smaller model for first-response tasks.
Red flags interviewers listen for
- Thinks decode is compute-bound.
- Load-tests with 200-token prompts for a 10k-token product.
- Can't name continuous batching or PagedAttention.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Correct boundness of each phase and the latency formula. |
| Modern serving | Continuous batching, PagedAttention, prefix caching, chunked prefill, disaggregation — and when each matters. |
| Measurement | Percentile SLOs per metric and production-shaped load tests. |
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Zhong et al., 2024paper
- vLLM documentation — vLLM project, 2026docs