Model quantization
Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.
A model's Parameters & weights are normally stored as 16-bit numbers. Quantization stores them with fewer bits, such as 8 or 4, a bit like saving a photo at lower quality. The model gets two to four times smaller and faster, with a small loss in accuracy that you must measure.
It is how large models fit on cheaper hardware or laptops. This is different from vector quantization used to compress embeddings in search.
Rounding prices to the nearest 10 cents instead of tracking fractions of a cent. Totals stay almost exactly right, and the ledger is much smaller — unless you are doing precise accounting.
Assuming quality loss is uniformly tiny because a general benchmark barely moved. Losses concentrate in precise tasks like maths, code and rare languages, which may be exactly your use case.
Formats as of Oct 2026: FP8 (E4M3) weights and activations on H100/H200 — the production default, near-lossless with per-tensor or per-channel scales; INT8 (LLM.int8(), SmoothQuant) on older GPUs; 4-bit weight-only via GPTQ or AWQ (calibration-based, group size 128) for single-GPU and local serving; FP4/NVFP4 and MXFP4 on Blackwell-generation GPUs (OpenAI shipped gpt-oss weights in MXFP4); GGUF k-quants for llama.cpp on CPUs and laptops. KV-cache quantization (FP8) is separate and halves the KV cache.
Why it speeds decode: decode is bandwidth-bound, so reading 4-bit weights instead of 16-bit cuts bytes per step ~4× — single-stream tokens/s rises 2–3× in practice. At large batch sizes decode becomes compute-bound and weight-only INT4's dequantisation overhead eats the gain; FP8 with FP8 tensor cores keeps it. Quality: 8-bit is near-lossless on most evals; 4-bit costs ~1 point on general chat but can cost several points on maths, code, long-context and low-resource languages; below 4 bits degrades fast. Quantization-aware training produces more robust low-bit models than post-training quantization.
FP8 weights + FP8 KV on Hopper-class GPUs; AWQ INT4 (group size 128) only where memory forces it and the sliced eval passes; never ship a quantized artifact without re-running the Golden set split by task type (numeric, code, multilingual, long-context).
Financial-filings assistant on Llama-3.1-70B. BF16: 140 GB, 2×H100. FP8: 70 GB, golden-set accuracy −0.2 points. AWQ INT4: ~38 GB, fits one 48 GB L40S; general Q&A −0.9 points, but numeric exact-match on table extraction −6.1 points (digit transpositions). Decision: FP8 on H100s for the numeric route, INT4 on L40S for the summarisation route.
For 'cut serving cost 2×', quantization is the first lever: FP8 halves memory with near-zero loss on Hopper GPUs. Then add the staff caveat: 'I find out where 4-bit hurts on my own eval, sliced by task type, not on a public leaderboard.' Expect: 'why doesn't INT4 always speed things up?' — at high batch decode is compute-bound and dequantisation costs cycles.
| Precision | 70B weights | Quality vs BF16 | Typical use |
|---|---|---|---|
| BF16 | 140 GB | Reference | Training, max quality |
| FP8 (E4M3) | 70 GB | Near-lossless | Default on H100/H200 |
| INT4 AWQ/GPTQ | ~38 GB | −1 pt general, more on maths/code | Single GPU, local |
| FP4 / MXFP4 | ~35-40 GB | Model-dependent | Blackwell GPUs |
| < 4-bit GGUF | < 30 GB | Clear loss | Laptops, experiments |
Numbers worth knowing
- 70B weights
- 140 → 70 → ~38 GBBF16 → FP8 → INT4
- Single-stream decode speed-up
- 2–3×INT4 weight-only vs BF16, small batch
- FP8 quality loss
- < 0.5 pointsmost general evals
Try the numbers
Defaults: a 70B GQA model (80 layers, 8 KV heads, head dim 128) in BF16, with ~85 GB left after FP8 weights on 2×H100. Ignores activation and fragmentation overhead (paged attention keeps the latter near zero).
How to decide
| When | Do this |
|---|---|
| Serving on H100/H200 | FP8 weights and KV by default. |
| Model must fit one smaller GPU | AWQ/GPTQ INT4 after a sliced eval; keep FP8 for routes that regress. |
| High-batch throughput serving | Prefer FP8 over weight-only INT4; INT4 gains vanish when compute-bound. |
Cost maths
70B BF16 on 2×H100 at an assumed $3/GPU-hour = $4,380/month. FP8 on 1×H100 = $2,190/month if one GPU's KV room meets your concurrency; INT4 on one L40S at an assumed $1.50/hour = $1,095/month. Savings are real only if the quality loss on your task does not create rework costing more.
Operating it
- GPU memory, not compute, limits concurrency → quantize weights and KV
- Any eval slice drops > 2 points after quantization → keep higher precision for that route
- Moving to Blackwell-class GPUs → evaluate FP4 variants
- Golden-set accuracy per slice (numeric, code, multilingual, long-context) after any precision change — alert on > 2 points
- Tokens/s per GPU at production batch size before vs after quantization
- Production numeric-validation failures (totals that don't add up) per 1k documents
When it goes wrong
Analysts reported wrong revenue figures in summaries of quarterly filings, all off by transposed or dropped digits.
The sliced golden set showed numeric exact-match down 6.1 points on the INT4 artifact while overall accuracy was down only 0.9.
The team had switched the whole deployment to AWQ INT4 based on a public chat benchmark, which had no numeric-extraction slice.
Kept INT4 for summaries, moved numeric extraction to FP8, and made per-slice thresholds a release gate for any precision change.
Related terms
Sources
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Frantar et al., 2022paper
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Lin et al., 2023paper
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Dettmers et al., 2022paper
- FP8 Formats for Deep Learning — Micikevicius et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.