Parameters & weights
The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.
Parameters, also called weights, are the numbers a Neural network learns during training. When someone says a model is '8B' or '70B', they mean it has 8 or 70 billion of these numbers.
More parameters let a model hold more patterns, but they also mean more memory and more computation every time it answers. A model's file is essentially these numbers saved to disk.
The sliders on the mixing desk. The song the desk can produce depends entirely on where every slider is set; a bigger desk can make richer music but needs a bigger room.
Treating parameter count as a quality ranking. A smaller model trained on more and better data, or with better post-training, routinely beats a larger one; always check on your own Golden set.
Weight memory = params × bytes per param: BF16 2 bytes, FP8/INT8 1 byte, INT4 about 0.5–0.6 bytes once per-group scales are included (Model quantization). Then add the KV cache (per user, per token) and 5–10% runtime overhead. Dense compute is about 2 × params FLOPs per token, and decode reads every weight once per step, so a 70B FP8 model moves 70 GB per decode step regardless of batch size.
For Mixture of experts (MoE) models separate total parameters (memory: DeepSeek-V3 671B, Llama 4 Maverick ~400B, Qwen3-235B-A22B 235B) from active parameters per token (compute: 37B, 17B, 22B). Parameter count is a weak predictor of quality on its own: training tokens, data quality and post-training move results as much (Scaling laws). Llama 3 8B was trained on 15T tokens, ~1,900 tokens per parameter, almost 100× the Chinchilla-optimal ratio.
Size every deployment as weights + KV(p95 context × target concurrency) + 10% overhead, and keep at least 30% of GPU memory for KV. Serve FP8 weights on H100/H200-class GPUs by default; keep BF16 only where an eval shows FP8 regresses.
Sizing for a self-hosted contract-review model on one H100 80 GB: Llama-3.1-70B in FP8 is 70 GB of weights, leaving ~8 GB for KV after overhead. At FP8 KV (160 KB/token) that is ~50k live tokens: six 8k-token contracts at once. The same model in BF16 (140 GB) needs two H100s with tensor parallelism; quantized to AWQ INT4 (~38 GB) it leaves ~35 GB for KV on one card, at a quality cost you must measure on clause extraction.
When asked 'can we self-host this model?', do weights first (params × bytes), then KV for your p95 context × concurrency, then 10% overhead, and only then name GPUs. A staff answer adds: 'parameters decide whether it fits; active parameters and bandwidth decide how fast it runs.' Expect the follow-up 'and for an MoE?': size memory by total, speed by active.
| Model | BF16 (2 B) | FP8 / INT8 (1 B) | INT4 (~0.55 B) |
|---|---|---|---|
| 8B | 16 GB | 8 GB | ~4.5 GB |
| 70B | 140 GB | 70 GB | ~38 GB |
| 405B | 810 GB | 405 GB | ~225 GB |
| 671B MoE (37B active) | 1,342 GB | 671 GB | ~370 GB |
Numbers worth knowing
- 70B weights
- 140 GB BF16 / 70 GB FP8 / ~38 GB INT4excluding KV and overhead
- Decode bytes moved per step
- = weight sizedense model; 70 GB per step at FP8
- Llama 3 8B tokens per parameter
- ~1,90015T training tokens; Chinchilla-optimal is ~20
- DeepSeek-V3 total vs active
- 671B vs 37Bmemory by total, compute by active
How to decide
| When | Do this |
|---|---|
| Weights fit with < 30% memory left for KV | Quantize to FP8, add a GPU, or cap max_model_len; do not ship at that margin. |
| Model is MoE | Size memory by total parameters and throughput by active parameters. |
| Choosing between an 8B and a 70B | Run both on your Golden set; pick the smallest model that meets the quality bar. |
Cost maths
GPU cost/month = GPUs × $/GPU-hour × 730. Assume H100 at $3/GPU-hour (illustrative, Oct 2026). 70B BF16 on 2×H100 = $4,380/month with ~12 GB for KV; 70B FP8 on 2×H100 = same cost with ~85 GB for KV (≈7× the concurrency); 70B FP8 on 1×H100 = $2,190/month but only ~6 concurrent 8k-token users. Cost per user served is what to compare, not cost per GPU.
Operating it
- KV cache usage > 85% at p95 with < 30% of memory reserved for it
- Request queue time p95 > 500 ms while GPUs show headroom on compute
- A smaller model passes the golden set within 1 point → downsize
- GPU KV cache usage > 85% for 5 minutes
- Preempted requests > 1% of total over 15 minutes
- Requests waiting in queue > 2 × running requests
When it goes wrong
A self-hosted 70B FP8 model passed staging, then at launch p95 latency hit 40 s and requests were preempted at ~20 concurrent users.
vLLM metrics: KV cache usage pinned at 100% and the preemption counter climbing within minutes of launch.
Capacity was sized by weights alone on one H100: 8 GB of KV room holds ~50k tokens, about six users at 8k-token prompts.
Moved to 2×H100 with tensor parallelism and FP8 KV cache, capped max_model_len at 16k for that tier, and added a sizing check (weights + KV + overhead) to the deployment review.
Related terms
Sources
- Scaling Laws for Neural Language Models — Kaplan et al., 2020paper
- The Llama 3 Herd of Models — Llama Team, Meta, 2024paper
- DeepSeek-V3 Technical Report — DeepSeek-AI, 2024paper
Check yourself
Three questions. Every option is explained once you answer.