Inference
Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.
Inference is using a trained model, as opposed to training it. Every time you send a message to a chatbot, the model runs inference: it reads your Prompt & system prompt and generates a reply, one Token at a time.
Training happens occasionally and is very expensive; inference happens on every request, so its speed and cost decide how fast and affordable an AI product feels.
Training is the years a chef spends learning; inference is cooking each order. Customers only ever experience the cooking, so that is where speed and cost per plate matter.
Optimising prompt length to fix slow responses when the real cost is long outputs. Decode is sequential, so output tokens usually dominate end-to-end latency.
Prefill processes all prompt tokens in parallel — compute-bound, ~2 × params FLOPs per token — and sets Time to first token (TTFT). Decode generates one token per step; each step reads every weight plus the KV cache from HBM, so it is memory-bandwidth-bound and sets inter-token latency (ITL). Floor for a 70B FP8 model on 2×H100 (3.35 TB/s each): 70 GB ÷ 6.7 TB/s ≈ 10.4 ms per step, ~96 tokens/s for a single stream, no matter how fast the tensor cores are.
Modern serving (vLLM, SGLang, TensorRT-LLM) stacks four ideas: PagedAttention (KV in fixed-size blocks, near-zero fragmentation), continuous Batching & throughput (sequences join and leave every step), chunked prefill (split long prompts into 2k–8k-token chunks interleaved with decodes so one 100k prompt doesn't stall everyone), and prefix caching (reuse KV for shared prefixes; RadixAttention in SGLang, automatic prefix caching in vLLM). At scale, prefill/decode disaggregation runs the two phases on separate GPU pools and ships KV between them (DistServe, Splitwise; NVIDIA Dynamo and llm-d implement it), so each pool is sized for its own bottleneck. Input tokens are priced below output tokens because prefill is parallel and decode is sequential.
vLLM or SGLang, FP8 weights, PagedAttention (default), chunked prefill on with max_num_batched_tokens 4k–8k, prefix caching on, max_model_len = p99 real context, streaming to clients. Disaggregate prefill/decode only when TTFT and ITL SLOs conflict at your scale (dozens of GPUs).
RAG over an internal wiki: 6,000-token prompt, 300-token answer, 70B FP8 on 2×H100. Prefill ≈ 2 × 70e9 × 6,000 = 8.4e14 FLOPs ≈ 0.4 s at realistic utilisation; decode at ~16 ms/token under load ≈ 4.8 s. Halving retrieved context saves ~0.2 s of TTFT; halving the answer saves ~2.4 s. Output length dominates end-to-end latency.
Decompose latency out loud: queue + prefill (prompt length) → TTFT; tokens × ITL (output length, batch, bandwidth) → total. Then name the lever per phase: prefix caching and shorter prompts for prefill; smaller/quantized models, speculative decoding and shorter outputs for decode; disaggregation when one phase's SLO keeps breaking the other's. Expect: 'why are output tokens more expensive?' — sequential, bandwidth-bound decode.
Numbers worth knowing
- Decode floor, 70B FP8 on 2×H100
- ~10.4 ms/token (~96 tok/s per stream)70 GB ÷ 6.7 TB/s, before KV reads
- H100 SXM HBM bandwidth
- 3.35 TB/sper GPU
- Prefill compute
- ≈ 2 × params FLOPs/tokenplus attention term at long context
- Chunked prefill chunk
- 2k–8k tokensvLLM max_num_batched_tokens
How to decide
| When | Do this |
|---|---|
| TTFT too high | Prefix caching, shorter prompts, less queueing; check prefill first. |
| Streaming feels slow (ITL) | Smaller or quantized model, Speculative decoding, lower concurrency per GPU. |
| Long prompts spike everyone's ITL | Enable chunked prefill; at scale, disaggregate prefill and decode. |
Cost maths
Self-host $/M output tokens = GPU $/hour ÷ (tokens/s × 3,600 × utilisation) × 1e6. 2×H100 at an assumed $3/GPU-hour = $6/hour; at 2,400 tok/s aggregate decode and 100% utilisation that is 8.64M tok/hour → $0.69/M; at a realistic 30% average utilisation → $2.31/M. Compare with an API at an assumed $15/M output — self-hosting wins only with steady load.
Operating it
- Queue time p95 > 500 ms → add replicas
- TTFT and ITL SLOs both failing under mixed long/short traffic → disaggregate or split pools by prompt length
- Average GPU utilisation < 25% → consolidate, or move the route to an API
- TTFT p95 and ITL p95 per route vs SLO
- Queue (waiting) requests > 2 × running requests for 5 minutes
- Prefix-cache hit rate on routes with shared system prompts < 70%
When it goes wrong
Chat users saw streaming stall for 2–3 seconds at random; p99 inter-token latency jumped from 30 ms to 2,400 ms several times an hour.
ITL p99 dashboard correlated with requests whose prompt exceeded 100k tokens (contract bundles uploaded by a few users).
Each long prompt was prefilled in one monolithic step, blocking every in-flight decode on that replica.
Enabled chunked prefill (max_num_batched_tokens 8,192), routed prompts > 32k to a dedicated long-context pool, and added ITL p99 by prompt-length bucket to the dashboard.
Related terms
Where it shows up
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024paper
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Zhong et al., 2024paper
- SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng et al., 2023paper
- vLLM documentation — vLLM project, 2026docs
Check yourself
Three questions. Every option is explained once you answer.