Batching & throughput
Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.
A GPU running an Large Language Model (LLM) can handle many requests at the same time almost as cheaply as one, because the expensive part, reading the model's Parameters & weights from memory, is shared. Batching groups requests together to take advantage of that.
The trade-off: bigger batches mean more total answers per second and lower cost, but each individual answer streams a bit slower. Batch APIs go further, giving discounts for jobs that can wait hours.
A bus versus a taxi. The bus carries forty people for little more fuel than one, but each passenger waits for stops; the taxi is fastest for you and most expensive per person.
Load-testing with uniform short prompts. Real traffic has long prompts and long outputs that fill the KV cache and collapse effective batch size, so capacity plans built on toy traffic fail in production.
Decode is memory-bandwidth-bound: each step streams all weights from HBM to produce one token per sequence. Batching B sequences reuses that read for B tokens, so total throughput rises almost linearly until compute or KV cache memory saturates. Continuous (iteration-level) batching, introduced by Orca (OSDI 2022) and standard in vLLM, SGLang and TensorRT-LLM, admits and retires sequences at every decode step, so short answers don't wait for long ones. Chunked prefill packs pieces of new prompts into the same step as ongoing decodes, holding inter-token latency steady while keeping the GPU busy (Sarathi-Serve).
The operating point is a curve: more concurrency lowers cost per token but raises inter-token latency (ITL) and queueing, pushing up p95. Name the metric: per-stream speed is tokens/s per user = 1 / ITL; total throughput is tokens/s per GPU. Capacity planning picks the highest concurrency whose ITL p95 and TTFT p95 still meet the SLOs on real prompt and output length distributions. Offline work (evals, backfills, enrichment) goes to provider batch APIs at ~50% off with up to 24-hour turnaround.
Continuous batching (default in vLLM/SGLang), chunked prefill on, max_num_seqs set from a load test at real token distributions, ITL p95 and TTFT p95 SLOs as the stopping rule, 30% headroom; all non-interactive jobs through provider batch APIs (~50% discount).
Load test, one H100 serving an 8B model, 1k-token prompts and 256-token outputs. Concurrency 1: ~90 tokens/s total, 11 ms ITL. Concurrency 32: ~1,780 tokens/s total, 18 ms ITL. Total throughput is ~20× higher, so cost per token falls ~20×. For each user, ITL rises 64% (11 → 18 ms), which means per-stream speed falls ~39% (91 → 56 tokens/s) — still well above reading speed. At concurrency 64, ITL reaches ~27 ms and total throughput ~2,400 tokens/s: the curve is flattening.
For 'how many GPUs for X requests at p95 Y?', walk it: real token distributions → tokens/s demand → the throughput-vs-ITL curve from a load test → highest concurrency meeting the SLO → replicas + 30% headroom. Staff line: 'Batching sells per-stream latency to buy throughput; the SLO tells me how much I can sell.' Expect: 'what goes wrong in production?' — long prompts and outputs fill KV and collapse effective batch size.
Numbers worth knowing
- Throughput gain, batch 1 → 32
- ~20× (91 → 1,778 tok/s)8B on 1×H100, this page's load test
- ITL change, batch 1 → 32
- +64% (11 → 18 ms)= ~39% fewer tokens/s per stream
- Batch API discount
- ~50%OpenAI Batch, Anthropic Message Batches; ≤ 24 h turnaround
How to decide
| When | Do this |
|---|---|
| Interactive route with an ITL SLO | Highest concurrency where ITL p95 meets the SLO on real traffic. |
| Work can wait hours (evals, backfills, enrichment) | Provider batch API, or a separate high-concurrency pool. |
| Long prompts keep stalling decodes | Chunked prefill; separate long-context pool. |
Cost maths
Cost per M output tokens = GPU $/hour ÷ (tokens/s × 3,600) × 1e6. One H100 at an assumed $3/hour: batch 1 → $3 ÷ (91 × 3,600) × 1e6 ≈ $9.16/M; batch 32 → ≈ $0.47/M; batch 64 → ≈ $0.35/M if the SLO allows 27 ms ITL. Batch API alternative for offline jobs: 50% off list price with no capacity to manage.
Operating it
- ITL p95 > SLO at current concurrency → lower max_num_seqs or add replicas
- Running sequences well below max_num_seqs while KV usage > 90% → memory-bound; FP8 KV or shorter contexts
- > 20% of tokens are offline jobs on the interactive pool → move them to batch
- ITL p95 and TTFT p95 per pool vs SLO
- Waiting vs running sequences — alert when waiting > running for 5 minutes
- KV cache usage > 90% with running sequences < 50% of max_num_seqs
When it goes wrong
A capacity plan said 4 GPUs would serve launch traffic; on day one p95 TTFT went from 0.8 s to 6 s and queues grew all afternoon.
Queue-time and KV-usage dashboards: KV pinned at 100% with only 9 running sequences per GPU instead of the planned 32.
The load test used 500-token prompts and 100-token answers; production RAG prompts averaged 6,000 tokens with 400-token answers, so KV filled at a third of the planned batch.
Re-ran load tests replaying sampled production prompts, re-sized to 11 GPUs, enabled FP8 KV and prefix caching, and made 'replay real traffic' a requirement for capacity sign-off.
Related terms
Where it shows up
Sources
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI '22) — Yu et al., 2022paper
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- Batch API — OpenAI, 2026docs
Check yourself
Three questions. Every option is explained once you answer.