How do you reduce latency in an agent?
- 2 min — latency model per step
- 3 min — levers ranked by impact
- 3 min — docs-assistant before/after numbers
- 2 min — fan-out tail maths and per-step tracing
- 2 min — SLO ownership and async split
Agent latency ≈ Σ over sequential steps of (model time + tool time). A model step is TTFT (0.5–1.5 s for a 20–40k-token prompt) plus output tokens at roughly 50–100 tokens/s, so a 250-token step costs ~3–5 s; tools add 0.1–2 s each. An 8-step run lands at 30–45 s. Levers, by impact: (1) fewer sequential model turns — each one removed saves ~4 s; (2) parallel tool calls in one turn, which Anthropic's 2024 taxonomy calls parallelization; (3) prompt caching, which cuts TTFT on long prefixes (Anthropic cites up to ~85% for long cached prompts); (4) smaller, faster models for routing and extraction steps; (5) streaming progress so perceived latency drops even when total doesn't.
Fan-out has a tail cost: with three independent calls, P(all finish under each one's p95) = 0.95³ = 86%, so the fan-out's p95 is each call's ~p98.
Developer-docs assistant answering “how do I rotate the signing key in SDK v4, and does the CLI support it?” v1: model calls search_docs, then search_changelog, then search_cli_reference, then answers — 4 model turns, p95 24 s. v2: the system prompt instructs issuing independent searches in one turn; three calls run concurrently (p95 of the fan-out 1.4 s vs 2.7 s sequential), and a small model writes the search queries. 2 model turns, p95 12 s, same rubric score on 150 eval questions. Long migrations (“upgrade my project to v4”) move to async: the user gets a progress link and a notification.
Track p50 and p95 end to end and per step, split into model time vs tool time vs queueing; the slow tail is usually one tool or a few runs with many steps. Budget latency like an SLO: interactive answers p95 ≤ 10 s, anything that can't meet it becomes a background job with progress — that is a product decision to make with PM up front, not an optimisation later. Speed changes need the same eval as quality changes: a smaller model or one fewer reflection pass that drops success 4 points is not a latency win.
Org scope. The product team owns the latency SLO and the interactive-vs-async split; the platform team owns per-step tracing, the parallel-execution runtime and model-routing infrastructure; tool-owning teams own per-tool p95 SLOs (e.g. search ≤ 500 ms). Build vs buy: provider “fast” tiers and regional endpoints are cheaper wins than self-hosting for latency. Failure mode: parallel calls that race on shared state (two writes to the same record) — parallelise reads freely, writes only when independent.
Assumptions: TTFT 0.8 s (30k prompt, cached), 250 output tokens at 70 tok/s ≈ 3.6 s → ~4.4 s per model turn; tools p50 0.4 s, p95 0.9 s. v1: 4 turns + 3 sequential tools ≈ 17.6 + 1.2 = 18.8 s p50. v2: 2 turns + one parallel fan-out ≈ 8.8 + 0.5 = 9.3 s p50. Fan-out p95 ≈ each tool's p98: 0.95³ = 0.857.
Sequential (v1)
- 4 model turns
- 3 tools in series
- p95 24 s
Parallel (v2)
- 2 model turns
- 3 tools concurrently
- p95 12 s, same rubric score
What the answer sounds like at each level
Run tool calls in parallel, use smaller models for simple steps, and stream progress.
Decomposes latency per step into TTFT, decode and tool time, removes sequential turns, accounts for fan-out tail maths, and validates speedups against the eval.
Sets an interactive latency SLO with PM, routes over-budget work to async, and splits ownership of per-tool SLOs, routing and tracing across teams.
Follow-ups the interviewer will push
1Now one tool has p95 of 6 s. Options?
Ask its owner for an SLO and a cheaper query shape; meanwhile cache its results per input with a TTL, prefetch it speculatively when the router predicts it will be needed, or move that path to async.
2Would a reasoning model make it slower?
Per step yes — thinking tokens are decoded and billed as output — but it may cut steps. Measure end-to-end p95 and success on the eval with a thinking budget cap; it's a win only if steps drop more than per-step time rises.
3How do you make a 40-second task feel fast?
Stream the plan and each completed step within 1–2 s, show partial results as they arrive, and let the user act on them. Perceived latency is time to first useful output.
Red flags interviewers listen for
- Optimises tool latency without measuring that model turns dominate.
- Ships a faster model with no eval comparison.
- Parallelises writes that touch shared state.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Latency model | TTFT + decode + tool time per step, summed over sequential turns. |
| Prioritisation | Removes turns before micro-optimising tools. |
| Tail awareness | Fan-out p95 maths and per-step p95 tracking. |
| Org scope | SLO owners, per-tool SLOs, product decision on async. |
Sources
- Building effective agents — Anthropic (Schluntz & Zhang), 2024blog
- Prompt caching — Anthropic, 2025docs