Speculative decoding
A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.
Big models generate text slowly because they produce one Token at a time. Speculative decoding speeds this up by letting a small, fast model guess the next few tokens, then having the big model check all those guesses at once.
When the guesses are right, which is common for predictable text, you get several tokens for the price of one big-model step. When they are wrong, the big model corrects them, so the final answer matches what the big model would have written anyway.
A junior drafts the next few sentences; the senior reviews them in one read, keeps the correct ones and rewrites from the first mistake. Same final document, far less senior time.
Assuming it always helps or that it lowers quality. It is distribution-preserving, but at high batch sizes or low acceptance the extra drafting can make throughput worse.
Draft k tokens with a cheap proposer, run the target model once over all k positions in parallel, accept the longest prefix that passes a rejection-sampling test, and at the first rejection sample a corrected token from an adjusted distribution. This provably preserves the target model's output distribution (Leviathan et al., arXiv 2022 / ICML 2023; Chen et al., 2023), so quality is unchanged — unlike quantization or distillation. With per-token acceptance rate α and k drafted tokens, expected tokens per target step = (1 − α^(k+1)) / (1 − α); speed-up = that ÷ (1 + k·c), where c is draft cost relative to a target step. Leviathan et al. reported 2–3× on T5-XXL.
Proposers: a small same-family model; extra decoding heads (Medusa); feature-level drafters (EAGLE family, the strongest general-purpose option in vLLM and SGLang); n-gram / prompt lookup (excellent when output copies input: code edits, RAG quoting, form filling); and native multi-token-prediction heads (DeepSeek-V3 reports 85–90% acceptance of the second token and ~1.8× tokens/s). It shines at low batch size, where decode is bandwidth-bound and the GPU has spare compute. At high batch the GPU is already compute-bound, so verifying rejected drafts wastes capacity and can lower throughput. The draft must share the target's tokenizer.
Enable on low-concurrency, latency-sensitive tiers only: prompt-lookup (n-gram) for code edits and RAG quoting, EAGLE-style drafters or built-in MTP heads for general chat; k = 3–5; measure acceptance rate and tokens/s per GPU in an A/B before rollout; disable on saturated batch pools.
Code-edit assistant rewriting a 200-line file with small changes. Prompt-lookup drafting proposes k = 8 tokens copied from the original file; α ≈ 0.85, so ~5.1 tokens are produced per target step, and single-stream decode rises from ~50 to ~130 tokens/s. For free-form brainstorming on the same model, α falls to ~0.4, expected tokens per step to ~1.7, and after draft overhead the gain is under 1.2×.
Staff-level check: explain why it is lossless (rejection sampling preserves the distribution), give the expected-tokens formula, and say when it stops helping (high batch, low acceptance, creative text). A strong line: 'It is a latency trick for low-concurrency tiers, not a throughput trick for saturated pools.' Expect: 'which drafter?' — prompt lookup for copy-heavy work, EAGLE-style for general chat, MTP if the model ships with it.
Numbers worth knowing
- Leviathan et al. speed-up
- 2–3×T5-XXL, identical outputs
- DeepSeek-V3 MTP
- 85–90% acceptance, ~1.8× tokens/ssecond-token prediction
- Expected tokens per step
- 3.36α = 0.8, k = 4
How to decide
| When | Do this |
|---|---|
| Output largely copies input (code edits, quoting) | Prompt-lookup drafting; k = 5–10. |
| Interactive chat at low concurrency | EAGLE-style drafter or native MTP; k = 3–5. |
| Pool runs at high batch near compute saturation | Leave it off; measure before enabling. |
Cost maths
Speed-up = (1 − α^(k+1)) / ((1 − α)(1 + k·c)). α = 0.8, k = 4, draft cost c = 0.05 → 3.36 / 1.2 = 2.8× per stream. At low concurrency that means the same GPU serves a latency SLO with ~2.8× fewer GPU-seconds per answer; at α = 0.4, k = 4, c = 0.05 → 1.65 / 1.2 = 1.4×, and at high batch the real number can drop below 1×.
Operating it
- ITL SLO missed on a low-concurrency premium tier → add speculative decoding before more GPUs
- Acceptance rate < 0.5 on a route → change drafter or disable
- Pool concurrency rises past the point where A/B tokens/s per GPU is ≤ baseline → disable
- Draft acceptance rate per route < 0.5
- Tokens/s per GPU with vs without speculation (A/B) — alert if ≤ baseline
- ITL p95 on the latency tier vs SLO
When it goes wrong
After speculative decoding was enabled fleet-wide, total tokens/s per GPU on the batch-enrichment pool fell 15% while the interactive pool got faster.
A/B of tokens/s per GPU by pool, plus the acceptance-rate metric (0.45 on enrichment prompts).
The enrichment pool ran at batch 64, already compute-bound; verifying rejected drafts consumed compute that would have produced accepted tokens.
Enabled speculation only on the interactive tier, set a per-pool flag tied to concurrency, and alerted when acceptance fell below 0.5.
Sources
- Fast Inference from Transformers via Speculative Decoding (ICML 2023) — Leviathan et al., 2022paper
- Accelerating Large Language Model Decoding with Speculative Sampling — Chen et al., 2023paper
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Cai et al., 2024paper
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Li et al., 2024paper
- DeepSeek-V3 Technical Report — DeepSeek-AI, 2024paper
Check yourself
Three questions. Every option is explained once you answer.