p50 / p95 / p99 latency
Latency at a percentile: p50 is the typical request, p95/p99 the slow tail. SLOs use the tail because averages hide the users who wait.
If you sort all request times, p50 is the middle one, p95 is slower than 95% of requests, and p99 slower than 99%. Averages are misleading because a few very slow requests barely move them. Users remember the slow ones, so systems are judged on p95 or p99.
A commute that takes 30 minutes on a typical day but 90 minutes one day in twenty. The average says 33; you leave home based on the 90.
Averaging p99s across pods or time windows. The mean of percentiles is not a percentile; aggregate the histograms and then compute.
Compute percentiles from histograms (Prometheus histograms, HDRHistogram, OpenTelemetry exponential histograms), never by averaging per-host percentiles, which is mathematically wrong. Tails compound with fan-out: if a request waits on n parallel calls each with a 1% chance of hitting its p99, the chance at least one is slow is 1 − 0.99^n: 9.6% at n = 10, 63% at n = 100 (Dean & Barroso, 'The Tail at Scale'). LLM systems have two latencies that need separate percentiles: Time to first token (TTFT) (prefill-bound, scales with prompt length) and inter-token latency or total generation time (decode-bound, scales with output length and batch load). A RAG request's p95 is not the sum of stage p95s, but each stage's tail feeds the end-to-end tail; budget per stage and use hedged requests, timeouts and fallbacks to cut it.
OpenTelemetry spans per stage with exponential histograms; dashboards show p50, p95, p99 for end-to-end, TTFT and each stage; SLO on p95 for interactive endpoints; per-intent max_tokens; timeouts at 2–3× the stage p99 with fallbacks.
Customer-support RAG endpoint, 24 h of traffic. End-to-end p50 = 1.6 s, p95 = 4.1 s, p99 = 9.8 s. Breakdown at p95: query rewrite 350 ms, hybrid retrieval 120 ms, rerank 280 ms, TTFT 900 ms, generation 2.3 s. The p99 traces are dominated by 1,200+ token answers and provider 429 retries. Fixes: cap answers at 350 tokens for this intent, add a 2 s hedge to a second region for TTFT. p99 drops to 5.2 s; p50 barely moves.
Say: I set SLOs on p95 or p99 per stage and end to end, split TTFT from total time for LLM calls, and use histograms so percentiles aggregate correctly. Expect: 'p99 doubled but p50 is flat, what do you check?' Long outputs, retries and rate limits, cold starts, and a single slow dependency in a fan-out.
Numbers worth knowing
- Fan-out tail probability
- 1 − 0.99^nn parallel calls, each with 1% chance of p99 latency; 63% at n = 100
- Chat TTFT target
- p95 < 1–1.5 sinteractive streaming UI
- Samples for a stable p99
- ≥ 10,000 per windowfewer and p99 is a handful of requests
How to decide
| When | Do this |
|---|---|
| Streaming chat UI | SLO on TTFT p95 plus total time p95 per intent |
| Fan-out to many shards or tools | Hedged requests and per-call timeouts; budget the tail per call |
| Low traffic (< 10k requests per window) | Use p95 and longer windows; p99 will be noise |
Cost maths
Hedging costs extra calls: hedge after the p95 delay → ~5% of requests send a duplicate. At $0.01 per LLM call and 1M requests/month, that is ~$500/month to cut the p99 roughly in half for the hedged stage. Cancel the loser on first token to keep output-token cost near zero.
Operating it
- p99/p50 ratio > 5 for an interactive endpoint
- TTFT p95 grows with prompt length after a context-size increase
- Provider 429 or retry share > 2% of calls
- End-to-end p95 and p99; alert when p95 exceeds SLO for 10 min
- TTFT p95 by prompt-length bucket; alert on > 30% regressions after deploys
- Retry and timeout rate per dependency; alert > 2%
When it goes wrong
Dashboards showed p99 at 3 s all month; users complained of 20-second waits
A user-side RUM trace contradicted the server dashboard; an engineer recomputed from raw spans
The dashboard averaged per-pod p99 values across 40 pods; a few overloaded pods' tails were diluted away
Switched to aggregated histograms; true p99 was 14 s, caused by uneven load balancing to pods with long-context sessions. Added least-outstanding-requests balancing
Related terms
Where it shows up
Sources
- The Tail at Scale — Dean & Barroso, 2013paper
- Site Reliability Engineering, ch. 4: Service Level Objectives — Google, 2016book
Check yourself
Three questions. Every option is explained once you answer.