Time to first token (TTFT)
Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.
Time to first token (TTFT) is how long a user waits before the answer starts appearing. With streaming, people start reading as soon as the first words show up, so TTFT matters more to perceived speed than how long the full answer takes.
It grows with longer prompts, busy servers and any steps before the model even starts, such as search or tool calls.
A waiter who brings bread within a minute versus one who brings everything at once after twenty. Total time may match, but the first one feels far faster.
Reporting only average end-to-end latency. Users feel p95 TTFT; averages hide tail queueing, long-prompt requests and buffering proxies.
TTFT = network + queueing + preprocessing (retrieval, reranking, tool calls) + prefill (+ thinking, for Reasoning models & extended thinking). After that, inter-token latency (ITL) governs streaming speed: end-to-end ≈ TTFT + output tokens × ITL. Prefill scales with prompt length, with a superlinear attention term at long context; queueing explodes as utilisation approaches saturation.
Levers by stage: Prompt caching of stable prefixes skips most prefill (Anthropic advertises up to 85% latency reduction on long prompts; cache reads cost 0.1× input); shorter or reordered prompts; parallel retrieval and reranking; smaller or quantized models; regional routing; and on self-hosted stacks, chunked prefill and prefill/decode disaggregation so long prompts don't block others. Measure TTFT on the client, not just the server: proxies that buffer server-sent events can turn a 0.9 s server TTFT into a full-response wait. Track p50 / p95 / p99 latency (p50/p95/p99) per route and set SLOs on it.
Stream every user-facing response; static instructions and documents first in the prompt with a cache breakpoint; retrieval and reranking in parallel where possible; client-side TTFT measurement (RUM) alongside server metrics; SLO of p95 TTFT < 1 s for chat and < 2.5 s for RAG.
Customer-support assistant, p95 TTFT 3.4 s: 600 ms retrieval, 350 ms reranking, 350 ms queueing, 2,100 ms prefill of a 14,000-token prompt. Moving the static 10,000-token system prompt and policy text to a cached prefix cuts prefill to ~500 ms and p95 TTFT to ~1.8 s; running retrieval and reranking in a tighter pipeline later saves another 300 ms. Answers are unchanged.
Decompose into stages, then name one lever per stage, and separate TTFT from throughput. A strong line: 'Users judge speed by the first token; I budget latency per stage and measure TTFT on the client, because proxies and reasoning both hide in that number.' Expect: 'what if the model is a reasoning model?' — show progress or a partial answer, or move reasoning off the interactive path.
Numbers worth knowing
- Chat TTFT target
- p95 < 1 sinteractive assistants
- RAG TTFT target
- p95 < 2.5 sincluding retrieval and reranking
- Prompt-caching latency gain
- up to 85%long cached prompts, Anthropic docs
- Comfortable streaming speed
- ITL < 50 ms (> 20 tok/s)faster than reading speed
How to decide
| When | Do this |
|---|---|
| Prefill dominates TTFT | Cache the stable prefix; trim retrieved context to what the eval needs. |
| Queueing dominates | Add capacity or shed load; see Backpressure. |
| Server TTFT is fine but users complain | Measure on the client; check proxy buffering and SSE flushing. |
Cost maths
Caching also cuts cost. 14,000-token prompt, 10,000 cached. Assume $3/M input; cache reads 0.1×, writes 1.25× (5-minute TTL). Uncached: $0.042 per call. Cached at a 95% hit rate: 10k × ($0.30 × 0.95 + $3.75 × 0.05)/M + 4k × $3/M ≈ $0.0047 + $0.012 = $0.017 — 60% cheaper and ~1.6 s faster at p95.
Operating it
- p95 TTFT > SLO for 3 consecutive days
- Prefill > 50% of TTFT on a route with a stable prefix → prompt caching
- Client-measured TTFT > 2× server-measured TTFT
- Client-side TTFT p95 per route vs SLO
- Server vs client TTFT gap > 500 ms
- TTFT stage breakdown (queue, retrieval, prefill) — alert when any stage's p95 grows > 30% week over week
When it goes wrong
Users said the assistant 'thinks for 8 seconds then dumps the whole answer', while server dashboards showed p95 TTFT of 0.9 s.
Client-side real-user monitoring added after the complaints: time to first rendered character matched full response time.
A new reverse proxy in front of the API buffered responses (default proxy buffering), so streamed tokens arrived all at once.
Disabled buffering for the streaming route (proxy_buffering off / X-Accel-Buffering: no), flushed SSE events per chunk, and made client-side TTFT the SLO metric.
Where it shows up
Sources
- Prompt caching — Anthropic, 2026docs
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Zhong et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.