Scaling laws
Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.
Scaling laws are patterns researchers measured: give a model more Parameters & weights, more Training data and more computing power, and its error goes down in a smooth, predictable way.
The catch is diminishing returns: each equal improvement costs many times more than the last. Labs use these curves to decide how big a model to build and how much data to feed it for a given budget.
Studying for an exam: the first 10 hours lift your score a lot, the next 100 hours lift it less, and the next 1,000 hours lift it a little more. The improvement is real and predictable, but each step costs far more.
Reading scaling laws as 'bigger is always better for my product'. They describe average loss for the trainer; your task may already be saturated by a smaller, cheaper model. Measure on your own evals.
Kaplan et al. (2020) fit loss as power laws in parameters N, data D and compute C over seven orders of magnitude. Hoffmann et al. (2022, Chinchilla) refit with a parametric form L(N, D) = E + A/N^0.34 + B/D^0.28 (E ≈ 1.69, A ≈ 406, B ≈ 411) and showed that for a fixed training budget N and D should grow together: about 20 tokens per parameter.
Production labs now train far past that ratio — Llama 3 8B saw ~1,900 tokens per parameter — because lifetime cost = training (6·N·D) + serving (2·N per token × every token ever served), and a smaller over-trained model is cheaper to serve. Since 2024 a second axis matters: test-time compute, where Reasoning models & extended thinking trade more thinking tokens at Inference for accuracy (Snell et al., 2024). Caveats: the laws predict average loss, not your task's accuracy; downstream skills can jump abruptly; and high-quality data, not FLOPs, is increasingly the binding constraint.
For product decisions, ignore the training curve and run your own scaling sweep: 2–3 model sizes × 2–3 reasoning efforts on your Golden set, plotted as accuracy vs cost per 1k requests. Choose the cheapest point within 1 point of the best.
Same training budget, two choices. 7.2e23 FLOPs buys Llama-3-8B-style over-training (8B on 15T tokens) or a compute-optimal ~77B model on ~1.5T tokens (6 × 77e9 × 1.55e12 ≈ 7.2e23). The 77B has slightly lower loss but costs ~10× more per served token. At 50B tokens served per day, that is the difference between ~2 and ~20 H100-class GPUs running continuously for the life of the model.
When asked 'bigger model or more thinking?', say there are two axes, both log-linear: training scale (bought once, paid back in serving) and test-time compute (paid per request). Recommend measuring the accuracy-vs-cost curve on your own eval for 2–3 model sizes and 2–3 reasoning efforts, then picking the cheapest point that meets the bar. Expect: 'what breaks the law?' — data exhaustion, contamination, and the gap between loss and task metrics.
| Model | Params | Tokens | Tokens/param |
|---|---|---|---|
| Chinchilla | 70B | 1.4T | 20 |
| Llama 3 70B | 70B | 15T | ~214 |
| Llama 3 8B | 8B | 15T | ~1,900 |
Numbers worth knowing
- Compute-optimal ratio
- ~20 tokens per parameterChinchilla, fixed training budget
- Chinchilla exponents
- α ≈ 0.34 (params), β ≈ 0.28 (data)Approach 3 fit
- Llama 3 8B
- ~1,900 tokens/paramdeliberately over-trained for cheap serving
How to decide
| When | Do this |
|---|---|
| Your eval saturates on a small model | Ship the small model; bigger buys nothing measurable. |
| Hard multi-step tasks fail on all sizes | Try more test-time compute (Reasoning models & extended thinking) before a larger base model. |
| You train your own model with heavy expected traffic | Over-train a smaller model past 20 tokens/param; serving dominates lifetime cost. |
Cost maths
Lifetime FLOPs = 6·N·D (train) + 2·N·T (serve T tokens). For N = 8B, D = 15T: train 7.2e23; serving 50B tokens/day for a year (T = 1.8e13) adds 2.9e23. A 77B compute-optimal alternative: same train cost, but serving becomes 2.8e24 — total 3.5e24 vs 1.0e24. Whenever the 8B meets the quality bar it wins, and the gap grows linearly with every token served.
Operating it
- Golden-set accuracy improves < 1 point per 2× model cost → stop scaling the model
- Cost per task is dominated by reasoning tokens → tune effort, not model size
- Domain loss still falling steeply with more domain data → collect data before buying compute
- Accuracy per $1k spend by route — alert if a model change lowers it > 10%
- Golden-set delta after any model upgrade < 1 point while cost rises > 50% → flag for review
- Share of traffic routed to the largest tier > target (e.g. 10%)
When it goes wrong
A team moved a document-classification route from a small model to a frontier model 'because bigger is better'; the bill rose 6× with no visible change.
Monthly cost review plus a re-run of the 500-item golden set: 94.1% vs 94.4%, inside the noise.
The task was saturated; scaling-law intuition about average loss was applied to a narrow classification task.
Reverted to the small model, added a cost-vs-accuracy sweep as a required artifact for any model upgrade, and routed only the 3% of low-confidence items to the large model.
Related terms
Sources
- Scaling Laws for Neural Language Models — Kaplan et al., 2020paper
- Training Compute-Optimal Large Language Models — Hoffmann et al., 2022paper
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Snell et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.