What are the trade-offs of quantizing a model?
- 1 min — what gets quantized and why it speeds decode
- 2 min — memory ladder for a real model
- 2 min — quality by task and method
- 2 min — rollout and ownership
Quantization stores weights (and optionally activations and the KV cache) at lower precision. Because decode is memory-bandwidth-bound, halving bytes per weight raises single-stream decode speed close to 2×. Ladder for a 70B model: BF16 140 GB (2×H100) → FP8 70 GB (1×H100, little KV headroom) → INT4 AWQ/GPTQ ~38 GB. Quality: FP8 (weights + activations on Hopper/Blackwell) is the safe default, within ~1 point on most task evals; INT4 weight-only varies by method and model — larger models tolerate it better; below 4 bits degrades sharply.
A team self-hosts a 70B model for internal-wiki summaries and contract-field extraction. BF16 needs 2 H100s; INT4 AWQ fits one with ~1.6× throughput. On their eval: summaries score 4.41 vs 4.43 (n=400, within noise); extraction of exact monetary amounts drops from 97.2% to 93.8% (n=600, significant). Decision: INT4 for summarisation, FP8 for extraction, FP8 KV cache on both.
- Evaluate per task: arithmetic, exact-value extraction, code and long-context retrieval degrade first; chat quality last.
- FP8 KV cache doubles long-context concurrency for a small, measurable quality cost; measure it at your p95 context length.
- Default ladder: FP8 first, INT4 AWQ only where memory or cost forces it and the task eval passes.
- Org scope: the inference platform publishes a quantization matrix (model × precision × task-eval score) so product teams pick from tested configs instead of re-running experiments. Rollout: canary the quantized pool at 5% with task-level metrics, not just latency.
- Where it breaks: assuming a hosted "same model" is served at the same precision across providers. Third-party hosts differ; re-run evals when you switch hosts.
70B params: BF16 2 bytes → 140 GB; FP8 1 byte → 70 GB; INT4 0.5 byte + scales → ~38 GB. On one 80 GB H100 at INT4, ~35 GB remains for KV: at FP8 KV 160 KB/token ≈ 220k live tokens ≈ 55 concurrent 4k sessions.
What the answer sounds like at each level
Knows quantization trades precision for memory and speed.
Uses FP8 by default, INT4 selectively, measures per-task impact and knows why decode speeds up.
Publishes a tested precision matrix as a platform product, gates rollouts on task metrics, and accounts for provider-side precision differences.
Follow-ups the interviewer will push
1Why does INT4 speed up decode but not always prefill?
Decode is bandwidth-bound, so fewer bytes per weight help directly. Prefill is compute-bound; weight-only INT4 must dequantize to compute, so gains there are small unless kernels use low-precision math.
2Now a regulated tenant needs reproducible outputs from a pinned model.
Pin the exact quantized artifact hash and engine version, not just the model name; a re-quantization is a model change and goes through the full eval gate.
Red flags interviewers listen for
- Picks precision from a generic benchmark without task evals.
- Believes quantization is free quality-wise at any bit-width.
- Ignores KV cache precision as a lever.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Links bandwidth-bound decode to bytes per weight. |
| Numbers | Correct memory per precision for a named model. |
| Judgement | Per-task decision with significance and staged rollout. |
Sources
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Frantar et al., 2022paper
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Lin et al., 2023paper