QPS (queries per second)
Queries per second: request rate a system handles. For LLMs, convert QPS to concurrency and tokens/s; that is what sizes GPUs and bills.
QPS counts how many requests arrive each second. It is the starting number for capacity planning: how many servers, how many GPUs, how much money. For LLM apps, each request lasts seconds and produces hundreds of tokens, so you also need to know how many requests are in flight at once and how many tokens per second that means.
A coffee shop counts customers per minute, but staffing depends on how long each order takes; ten customers a minute for espresso is not ten a minute for pour-over.
Sizing LLM capacity from QPS alone. 10 QPS of 200-token chats and 10 QPS of 50k-token document reviews differ by 100× in prefill compute and cost.
Little's law: concurrency = arrival rate × time in system. 20 QPS × 6 s average generation = 120 concurrent streams. Token throughput = QPS × (input tokens for prefill, output tokens for decode); decode is the scarce resource because it is memory-bandwidth bound (KV cache). Size for peak, not average: consumer traffic peaks at 2–4× the daily mean, and agent workloads multiply QPS by the number of LLM calls per task (5–30). Provider limits are usually expressed as requests per minute (RPM) and tokens per minute (TPM), so translate: 20 QPS = 1,200 RPM; at 3k input tokens each = 3.6M input TPM. Self-hosted, the question becomes tokens/s per GPU at your target inter-token latency under continuous Batching & throughput (vLLM, SGLang).
Capacity sheet per endpoint: daily volume, peak-hour share, LLM calls per request, input and output tokens per call, end-to-end latency → peak QPS, concurrency, TPM, $/day. Load-test at 1.5× projected peak with production-shaped prompts (e.g. k6 or Locust replaying sampled requests).
Legal-contract Q&A, 400k queries/day, peak hour carries 12% of daily traffic. Average QPS = 4.6; peak = 400,000 × 0.12 / 3,600 = 13.3 QPS. Each query: 6k input, 500 output tokens, 7 s end to end. Peak concurrency = 13.3 × 7 ≈ 93 streams. Peak tokens: 80k input tokens/s and 6.7k output tokens/s, i.e. 4.8M input TPM: check that against the provider tier before launch.
Say: I convert QPS into concurrency with Little's law and into input/output tokens per second, size for peak hour, and multiply by LLM calls per request for agents. Expect: 'now 10× traffic': whichever of TPM limits, KV-cache memory or rerank GPU saturates first sets the next step; usually it is provider TPM, so request a limit increase or spread across regions.
| Quantity | Formula | Value |
|---|---|---|
| Peak QPS | 400k × 12% / 3,600 | 13.3 |
| Concurrency | QPS × 7 s | ≈ 93 streams |
| Input tokens/s | 13.3 × 6,000 | 80k |
| Output tokens/s | 13.3 × 500 | 6.7k |
| Input TPM | 80k × 60 | 4.8M |
Numbers worth knowing
- Peak-to-average ratio
- 2–4×consumer and business-hours traffic; batch jobs excluded
- LLM calls per agent task
- 5–30tool-using agents; multiply user QPS by this
- Concurrency
- QPS × seconds per requestLittle's law, steady state
How to decide
| When | Do this |
|---|---|
| Peak TPM > 70% of provider limit | Request a limit increase, add a second region or provider via the gateway |
| Load is bursty and latency-tolerant | Queue with backpressure or batch API (~50% discount) |
| Sustained high output tokens/s on one model | Price self-hosting on vLLM/SGLang against API cost |
Cost maths
Daily cost = queries/day × (input tokens × $in + output tokens × $out). Example above at $3/$15 per M (illustrative, Oct 2026): 400k × (6,000 × $3/M + 500 × $15/M) = 400k × ($0.018 + $0.0075) = $10,200/day ≈ $306k/month. Prompt caching a 4k static prefix at 0.1× cuts input cost by ~60%.
Operating it
- Peak TPM or RPM > 70% of the provider limit
- 429 rate > 0.5% at peak
- Concurrency at peak exceeds the connection pool or GPU batch capacity
- Peak QPS, RPM and TPM vs provider limits; alert at 70%
- 429 and queue-wait rates; alert > 0.5%
- LLM calls per user request (agents); alert on a > 30% rise
When it goes wrong
A newly launched research agent returned 429 errors for half of users at 9 a.m. on launch day
Gateway error-rate alert; provider dashboard showed TPM pinned at the limit
Capacity planning used user QPS (3/s) but each task made ~18 LLM calls with growing context; real TPM was 14× the estimate
Recomputed with calls-per-task, requested a tier increase, added queueing with visible progress, and capped agent steps at 12
Related terms
Where it shows up
Sources
- Site Reliability Engineering, ch. 4: Service Level Objectives — Google, 2016book
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.