Estimate tokens/s per GPU and the number of H100s needed to serve 50 QPS of a 70B model with 1,500-token prompts and 300-token answers.
- 2 min — clarify model, precision, prompt/output lengths, SLO
- 4 min — decode from bandwidth, prefill from FLOPs
- 2 min — headroom, N+1, cost
- 2 min — levers (prefix cache, chunked prefill, speculative decoding) and what the load test must confirm
Split the work into prefill (compute-bound: ~2 × params FLOPs per prompt token) and decode (memory-bandwidth-bound: every step reads all weights plus the batch's KV cache). Size each separately, add them, then add headroom for p99 and a failed replica. Most candidates size decode only and are surprised that long prompts dominate.
The number in the interview is a planning estimate; the number in the budget comes from a load test with the real prompt-length distribution. Name the levers that move it most: prefix caching on the shared 1,000-token instruction block (cuts prefill ~60%), chunked prefill so long prompts don't stall decodes, and speculative decoding for decode-heavy traffic. Organisationally, the platform team owns the fleet and charges back per million tokens; the app team owns the prompt length, which is the biggest cost driver they control.
Assumptions: Llama-3.3-70B FP8 (~70 GB weights), tensor-parallel 2 on H100 SXM (3.35 TB/s each), ~70% achievable bandwidth, batch 64 per replica, average live context 1,800 tokens, FP8 KV 160 KiB/token, FP8 dense peak ~1,979 TFLOPS/GPU at 50% MFU for prefill.
- Demand: decode 50 × 300 = 15,000 output tok/s; prefill 50 × 1,500 = 75,000 prompt tok/s.
- Decode per replica: weights 70 GB ÷ (2 × 3.35 × 0.7 ≈ 4.7 TB/s) ≈ 15 ms + KV reads (64 × 1,800 × 160 KiB ≈ 19 GB) ≈ 4 ms → ~19 ms/step → 52 steps/s × 64 ≈ 3,300 tok/s; plan on 3,000. → 15,000 ÷ 3,000 = 5 replicas = 10 GPUs.
- Prefill: 75,000 × 2 × 70 × 10⁹ = 1.05 × 10¹⁶ FLOP/s ÷ ~1 × 10¹⁵ effective per GPU ≈ 11 GPU-equivalents.
- Total: ~21 GPUs busy → at 75% target utilisation plus one spare replica → ~30 GPUs (15 TP2 replicas).
- With prefix caching on a 1,000-token shared prefix (60% of prefill gone): prefill ≈ 4.5 GPUs → ~15 busy → ~22 GPUs.
- Cost: assume $2.50/GPU-hour (illustrative reserved price, Oct 2026): 30 GPUs ≈ $55k/month; 22 GPUs ≈ $40k/month.
| Stage | Demand | Per-GPU capacity | GPUs |
|---|---|---|---|
| Decode | 15,000 tok/s | ~1,500 tok/s | 10 |
| Prefill | 75,000 tok/s | ~7,000 tok/s | 11 |
| Prefill w/ prefix cache | 30,000 tok/s | ~7,000 tok/s | 4.5 |
| Headroom + N+1 | — | — | +8 |
What the answer sounds like at each level
Computes output tokens/s needed (15,000) and divides by a per-GPU throughput figure, arriving at a plausible GPU count.
Derives decode throughput from memory bandwidth and batch size, sizes prefill from FLOPs, and adds headroom and N+1.
Shows which lever cuts the bill most (prefix caching), separates planning from load-tested numbers, and assigns ownership of prompt length and fleet chargeback.
Follow-ups the interviewer will push
1Now traffic is 10× (500 QPS).
Linear in GPUs (~220 with prefix caching), so the conversation shifts: disaggregate prefill and decode pools so each scales on its own metric, consider a smaller distilled model for the 70% of easy requests behind a router, and negotiate committed capacity — at that scale a 20% price difference is ~$1M/year.
2Now the SLO is p95 TTFT < 400 ms.
A 1,500-token prefill takes ~150 ms of pure compute on a TP2 replica; queueing is what breaks 400 ms. Run utilisation nearer 50%, cap max_num_batched_tokens with chunked prefill, and route by prefix to keep cache hits high — expect ~30–40% more GPUs.
3Would a mixture-of-experts model change the maths?
Yes: MoE decode reads only active experts per token, so FLOPs per token drop, but all weights must still sit in HBM and expert routing at large batch touches most experts. Memory sizing follows total params; compute follows active params.
Red flags interviewers listen for
- Sizes only decode and ignores prefill, under-provisioning by half for long-prompt workloads.
- Quotes a vendor's peak tokens/s as if it were sustained under an SLO.
- No headroom and no N+1 — the first replica failure breaks the SLO.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Decomposition | Separates prefill and decode and explains why one is compute-bound and the other memory-bound. |
| Numbers | Lands in the 20–35 GPU range with every assumption stated. |
| Levers | Quantifies prefix caching and SLO tightening rather than listing them. |
| Ownership | Says who owns prompt length vs fleet, and that the budget number comes from a load test. |
Sources
- Efficiently Scaling Transformer Inference — Pope et al., 2022paper
- Scaling Laws for Neural Language Models — Kaplan et al., 2020paper
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper