Debugging drill: p99 time-to-first-token jumped from 0.9 s to 4.5 s right after a deploy. Walk through it.
- 1 min — mitigate: rollback or flag
- 2 min — list symptoms precisely (p50 vs p99, ITL, errors)
- 3 min — ranked hypotheses and the splitting metric
- 2 min — fix and prevention: canary metric, template lint, ownership
Symptoms: p99 TTFT 5×, p50 only 1.3×, inter-token latency flat, GPU count unchanged, error rate flat. Hypotheses ranked: (1) prefix-cache hit rate collapsed because the prompt template changed (dynamic content moved ahead of the static prefix); (2) a scheduler change — max_num_batched_tokens or chunked prefill disabled — lets long prefills block the queue; (3) KV pressure from a raised max_model_len, causing queueing and preemption; (4) prompts got longer (new retrieval depth or tool schemas); (5) a cold or mis-sized replica behind the load balancer.
The one metric that splits them: decompose TTFT into queue time vs prefill time per request, alongside the prefix-cache hit rate vLLM exports. Queue time up with cache hit rate down → (1). Queue time up with hit rate flat and preemptions up → (3). Prefill time up per token with hit rate flat → (2) or (4); check prompt-token p99. One replica's TTFT diverging → (5). Fix for the common case: restore the static-first template (system prompt and tool schemas before anything per-request, timestamps last), roll back via the flag first.
Prevention: a canary gate on p99 TTFT and prefix-cache hit rate, not just error rate; a lint in CI that fails if the first N tokens of the rendered template contain request-variable fields; and prompt-template changes treated as deploys with versioning. Ownership: the app team owns the template, the serving team owns the cache, so the hit-rate metric must be visible to both in the same dashboard — that's how this class of incident stops recurring.
Why a cache miss explains 5×: Llama-3.3-70B FP8 on TP2 (~2 PFLOP/s effective prefill). Prompt 6,000 tokens of which 5,000 is a shared prefix. Cached: prefill 1,000 × 140 GFLOP ≈ 0.07 s. Uncached: 6,000 × 140 GFLOP ≈ 0.42 s — 6× compute per request. At 70% utilisation that pushes the replica past saturation; queueing theory (M/M/1 wait ∝ ρ/(1 − ρ)) turns 6× prefill into far worse p99 as ρ approaches 1. Expected signatures: hit rate 80% → <5%, prompt-token throughput per GPU flat, queue-time p99 up ~10×.
What the answer sounds like at each level
Rolls back, then checks GPU utilisation and prompt lengths.
Splits TTFT into queue vs prefill, checks prefix-cache hit rate and preemptions, finds the template change.
Adds the canary metric and template lint, assigns cross-team ownership of the cache-hit metric, and writes the postmortem around detection.
Follow-ups the interviewer will push
1Now the same symptom appears with no deploy.
Look for traffic mix shift: a new tenant with very long prompts, or a client sending unique per-user preambles. Per-tenant prompt-length p99 and per-tenant cache hit rate split it.
2Now only one region is affected.
Region-local cause: a smaller pool after a node drain, a different GPU SKU, or a regional cache warm-up after restart. Compare replica counts and the per-replica hit rate in that region against the others.
Red flags interviewers listen for
- Debugs in production before rolling back.
- Adds GPUs without explaining why compute per request rose.
- Treats TTFT as one number instead of queue + prefill.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Method | Symptoms → ranked hypotheses → one splitting metric → fix → prevention, in that order. |
| Domain depth | Knows prefix caching requires a byte-identical prefix and that dynamic fields break it. |
| Prevention | Proposes a CI lint and a canary gate on the right metric, with owners. |
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- vLLM documentation — vLLM project, 2026docs