Mixture of experts (MoE)
An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.
A mixture-of-experts (MoE) model contains many smaller sub-networks called experts. For each Token, a small router picks only a few of them to do the work, for example 8 out of 256.
This lets a model have a very large total number of Parameters & weights, and so hold a lot of knowledge, while each token uses only a fraction of them, which keeps it faster and cheaper to run than a dense model of the same total size.
A large hospital with many specialists. Each patient sees only the two or three relevant doctors, so the hospital has huge total expertise without every doctor examining every patient.
Thinking an MoE with 17B active parameters runs on hardware sized for a 17B model. Every expert must be resident in memory; you size memory by total parameters.
In MoE transformers the MLP in most blocks becomes E expert MLPs plus a learned router choosing the top-k per token, sometimes with always-on shared experts. Compute per token tracks active parameters; memory tracks total, because every expert must be resident. Reference points: Mixtral 8x7B — 46.7B total, 12.9B active (top-2 of 8); DeepSeek-V3 — 671B total, 37B active (1 shared + top-8 of 256 routed experts); Qwen3-235B-A22B; Llama 4 Maverick — ~400B total, 17B active; gpt-oss-120b — 117B total, 5.1B active, fits one 80 GB GPU in MXFP4.
Hard parts: load balancing (auxiliary losses, or DeepSeek-V3's auxiliary-loss-free bias adjustment) so the router doesn't collapse onto a few experts; serving with expert parallelism across GPUs, where all-to-all token exchange can dominate step time; and batch size — at low batch each expert sees few tokens, so the memory for all experts is paid while compute is idle. MoE pays off with enough memory and traffic to keep experts busy. Experts specialise in token-level patterns (punctuation, code syntax, languages), not neat topics like 'the legal expert'.
Self-host MoE only with steady, high-batch traffic: vLLM or SGLang with expert parallelism, FP8 (or the model's native MXFP4/FP8) weights, expert load balancing enabled, and per-rank step-time monitoring. At low or spiky volume, use a hosted API or a dense model of similar active size.
Sizing DeepSeek-V3 for an internal code-search assistant: 671 GB of FP8 weights before KV. 8×H100 (640 GB) cannot hold it; 8×H200 (8 × 141 = 1,128 GB) holds weights plus ~400 GB for KV, or two 8×H100 nodes with expert parallelism. Per-token compute is that of a ~37B dense model, so at high batch it serves far more tokens/s per GPU than a 405B dense model — but at 3 requests per minute, a single-GPU dense 32B model is cheaper.
Distinguish total vs active and explain what each sizes: 'total sets memory and GPU count; active sets FLOPs per token; batch size decides whether you get the benefit.' Expect: 'why is MoE serving harder?' — expert parallelism, all-to-all communication, hot experts, and needing enough traffic to fill every expert's batch.
| Model | Total | Active | Experts used |
|---|---|---|---|
| Mixtral 8x7B | 46.7B | 12.9B | top-2 of 8 |
| DeepSeek-V3 | 671B | 37B | 1 shared + top-8 of 256 |
| Qwen3-235B-A22B | 235B | 22B | top-8 of 128 |
| gpt-oss-120b | 117B | 5.1B | top-4 of 128 |
Numbers worth knowing
- DeepSeek-V3
- 671B total / 37B active1 shared + 8 of 256 routed experts
- Mixtral 8x7B
- 46.7B total / 12.9B activetop-2 of 8
- gpt-oss-120b
- fits one 80 GB GPUMXFP4 weights, 5.1B active
How to decide
| When | Do this |
|---|---|
| High steady traffic and multi-GPU nodes | MoE: best quality per FLOP served. |
| Single GPU or low traffic | Dense model with similar active parameters, or the API. |
| One GPU rank's step time is consistently slowest | Rebalance or replicate hot experts. |
Cost maths
GPU count = ceil((total params × bytes + KV + overhead) / GPU memory). DeepSeek-V3 FP8: 671 GB + 300 GB KV + 50 GB ≈ 1,021 GB → 8×H200 at an assumed $4/GPU-hour = $23,360/month. Worth it only if the node is busy: at 5,000 tokens/s sustained that is ~13B tokens/month ≈ $1.8/M; at 200 tokens/s it is ~$45/M.
Operating it
- Sustained traffic high enough to keep per-expert batches > 8 tokens per step
- All-to-all time > 30% of decode step → change parallelism layout or node topology
- Expert load imbalance (max/mean tokens per expert) > 2×
- Max/mean tokens per expert per step > 2×
- Per-rank step time spread > 1.5×
- All-to-all communication share of step time > 30%
When it goes wrong
On an expert-parallel MoE deployment, p99 ITL spiked every afternoon while average GPU utilisation looked low.
Per-rank step-time metrics showed one rank consistently 2.5× slower than the others during peak.
Afternoon traffic was dominated by code requests; the router sent a disproportionate share of tokens to experts that happened to live on one rank, which became the straggler every step waited for.
Enabled expert replication and load balancing (DeepSeek's open-source EPLB approach: duplicate hot experts across ranks), and alerted on per-rank imbalance.
Related terms
Sources
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Shazeer et al., 2017paper
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus et al., 2021paper
- Mixtral of Experts — Jiang et al., 2024paper
- DeepSeek-V3 Technical Report — DeepSeek-AI, 2024paper
Check yourself
Three questions. Every option is explained once you answer.