Explain speculative decoding and mixture-of-experts.
- 2 min — speculative decoding mechanism and acceptance math
- 2 min — MoE routing, total vs active params
- 2 min — when each helps or hurts
- 1 min — who should own it
Speculative decoding: a cheap drafter (small model, extra heads like Medusa/EAGLE, or n-gram lookup from the prompt) proposes γ tokens; the target model scores all of them in one forward pass and accepts the longest prefix consistent with its own distribution via rejection sampling, so output distribution is unchanged. With per-token acceptance α and γ drafted tokens, expected tokens per target pass = (1 − α^(γ+1)) / (1 − α): α=0.8, γ=4 gives 3.4. Mixture-of-experts: each layer has many feed-forward experts and a router sends each token to the top-k (often 2–8). Mixtral 8x7B: 46.7B total, 12.9B active per token; DeepSeek-V3: 671B total, 37B active. Compute per token tracks active params; memory tracks total params.
A code-completion service: code is predictable, acceptance α≈0.8, and speculative decoding with γ=4 cuts ITL from 30 ms to ~13 ms at batch ≤8. A general chat service hosts an MoE model to get large-model quality at ~37B-dense compute per token — but must hold all 671B parameters (~700 GB in FP8) across a node of 8×H200 or more.
- Speculative decoding helps most when the GPU is underused (small batches, latency-sensitive). At batch 64+ the GPU is already busy and gains shrink toward 1.1–1.3×; some engines disable it above a batch threshold.
- MoE saves compute, not memory, and adds expert-parallel communication and load-imbalance issues; serving needs high-bandwidth interconnect.
- Both are invisible behind hosted APIs; they explain why a provider's price and latency look the way they do.
- Org scope: the inference platform team owns these; application teams should only care via measured ITL and $/M tokens. Buy unless you run >20 GPUs of steady traffic.
- Where it breaks: low acceptance on unpredictable text (creative writing, low-resource languages) — α=0.4 gives ~1.6 tokens per pass, often slower than no speculation after draft cost.
α=0.8, γ=4: (1 − 0.8^5)/(1 − 0.8) = (1 − 0.328)/0.2 = 3.36 tokens per target pass. If one target pass = 30 ms and drafting 4 tokens = 6 ms: 36 ms / 3.36 ≈ 10.7 ms per token vs 30 ms → ~2.8× at small batch. α=0.4: 1.65 tokens/pass → 21.8 ms → ~1.4×.
What the answer sounds like at each level
Describes draft-and-verify and that MoE activates a subset of experts.
Computes expected accepted tokens from α and γ, and explains MoE compute vs memory with real model sizes.
Knows when each pays off at a given batch size and hardware, and frames them as platform decisions measured in ITL and $/M.
Follow-ups the interviewer will push
1Does speculative decoding change outputs?
With correct rejection sampling, no — the output distribution equals the target model's. Greedy variants match greedy output exactly.
2Why might an MoE model be cheaper per token from a provider but hard for you to self-host?
Providers amortise the full memory footprint across huge batches; you need all experts resident and fast interconnect even at low traffic, so your utilisation and $/token are worse.
Red flags interviewers listen for
- Thinks speculative decoding trades quality for speed.
- Says MoE reduces memory requirements.
- Doesn't relate speculation gains to batch size.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Rejection-sampling verification; top-k routing. |
| Quantitative | Accepted-tokens formula and total/active params for a real model. |
| Judgement | Batch-size dependence and hosting implications. |
Sources
- Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022paper
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus et al., 2021paper
- Mixtral of Experts — Jiang et al., 2024paper