Design ChatGPT-style chat serving for 10M daily users. How many GPUs do you need, and how do you get from QPS to that number?
- 5 min — scope: DAU, messages/user, tokens in/out, TTFT and inter-token SLOs, model size
- 10 min — estimate: QPS → output tok/s → concurrency → KV check → GPUs → $
- 15 min — architecture: gateway, conversation store, prefix router, engine pool, streaming, moderation
- 10 min — overload, failure and degradation; long-context pool
- 5 min — ownership, build vs buy, quarterly capacity review
Serving is two workloads glued together. Prefill processes the whole prompt once; it is compute-bound and sets TTFT. Decode emits one token per step per stream; it is memory-bandwidth-bound and sets inter-token latency. Capacity is therefore not QPS but output tokens per second at the latency SLO, bounded by KV-cache memory for concurrent streams.
The engine (vLLM or SGLang as of Oct 2026) gives you continuous batching, paged KV, chunked prefill and prefix caching. Your design is the fleet around it: gateway, rate limits, prefix-aware routing, autoscaling and a degradation ladder.
One message, end to end: the client opens an SSE stream → the gateway authenticates, applies a per-user token bucket and picks a model tier → the conversation service loads history and assembles system prompt + tools + history inside a context budget → the router sends it to a replica already holding that conversation's prefix in KV → the engine streams tokens → output moderation runs on the stream in parallel → the turn is persisted asynchronously.
Day-one defaults: a 70B-class model in FP8 on 8×H100 nodes, TP=4 per replica (two replicas per node), max_model_len at the real p99 context (32k, not the model maximum), chunked prefill on, FP8 KV cache. SLOs: TTFT p95 ≤ 800 ms, inter-token p95 ≤ 50 ms.
- Price by the scarce resource. A 100k-context user holds ~50× the KV of a 2k chat user. Quota and route long context to its own pool, or one tenant causes noisy-neighbour preemption storms for everyone.
- Degrade, don't fail. Under overload: cap max output, route the free tier to a smaller distilled model, queue with an honest ETA, then shed — in that order, each behind a flag that on-call can flip.
- Failure modes: a replica crash drops ~170 live streams (clients resume from the last persisted token); a viral system prompt hot-spots one replica under prefix routing; a GPU Xid error silently halves a node's throughput.
- Org scope: the serving-platform team owns engines, router, quotas and the capacity model; product teams own prompts and receive token budgets. Adopt the open-source engine and build the router and capacity tooling — do not fork vLLM unless you can staff a rebase every release.
Assumptions (illustrative, Oct 2026): 10M DAU × 15 messages = 150M requests/day; average prompt 1,500 tokens (~70% prefix-cached), 400 output tokens; 70B-class model with FP8 weights and KV; replica = 4×H100 (TP=4); each stream decodes at ~30 tok/s; $2.50 per H100-hour reserved.
- QPS: 150M / 86,400 ≈ 1,736 average; peak 3× ≈ 5,200 QPS.
- Output tokens/s at peak: 5,200 × 400 ≈ 2.1M tok/s.
- Concurrency (Little's law): a request lives ~0.5 s TTFT + 400/30 ≈ 14 s → 5,200 × 14 ≈ 73k live streams.
- KV check: ~160 KB/token × ~1,900 tokens ≈ 300 MB per stream. A replica has ~250 GB free after weights → ~800 streams by memory, but the 30 tok/s target caps it near 170 streams (≈5,100 tok/s per replica, ≈1,275 tok/s per GPU). Latency binds before memory.
- GPUs: 2.1M / 1,275 ≈ 1,650; +30% for failover and skew ≈ 2,150 H100s ≈ 270 nodes.
- $: 2,150 × 730 h × $2.50 ≈ $3.9M/month. Output = 150M × 30 × 400 = 1.8T tokens/month → ≈ $2.2 per M output tokens of GPU cost. Each +10 points of prefix-cache hit rate is worth roughly 3–5% of the fleet.
What the answer sounds like at each level
Puts a load balancer in front of GPU servers running vLLM, streams responses, and adds GPUs when latency rises.
Separates prefill from decode, sizes by output tokens/s at an SLO, uses Little's law for concurrency, checks KV memory, and adds prefix caching and rate limits.
Derives the GPU count from stated assumptions, prices long context separately, designs the degradation ladder and the capacity-planning loop, and names who owns router, quotas and model choice across teams.
Follow-ups the interviewer will push
1Traffic goes 10× in a week after a launch. Reserved GPUs cannot arrive that fast.
Run the degradation ladder: cap output at 1k tokens, route the free tier to an 8B–30B model, enable speculative decoding (1.5–2.5× decode speed at low batch, less at high batch), push prefix-cache hit rate, and burst to an API provider through the gateway with fallback. Sign the reservation in parallel.
2Users say answers stall mid-stream at peak, but TTFT looks fine.
That is inter-token latency: long prefills are blocking decode steps. Cap prefill tokens per step with chunked prefill (Sarathi-Serve style); at larger scale disaggregate prefill and decode pools (DistServe pattern, as in NVIDIA Dynamo or llm-d). Alert on inter-token p95 > 60 ms.
3One region loses its GPUs.
Run each of three regions at ≤70% of its peak so N-1 absorbs the loss; fail over at DNS/anycast; replicate conversation state so streams resume. Data-residency tenants fail over only inside their boundary.
Red flags interviewers listen for
- Sizes GPUs from QPS alone, without output tokens, context length or a latency target.
- Ignores KV-cache memory, so concurrency per GPU is unbounded in the design.
- Autoscales on GPU utilisation, which reads ~100% at almost any load.
- The only overload response is 'add GPUs' — no degradation ladder.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Estimation | States assumptions, applies Little's law, checks KV memory, lands within 2× of a defensible number. |
| Serving mechanics | Distinguishes prefill and decode; explains continuous batching, chunked prefill and prefix caching and what each buys. |
| Reliability | Degradation ladder, stream resumption, per-tier quotas, regional N-1. |
| Org and cost | Quotes $ per M output tokens, names router/quota ownership and the build-vs-buy line. |
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Zhong et al., 2024paper