Pretraining
The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.
Pretraining is how a language model first learns anything. It reads an enormous pile of text (web pages, books, code) and practises one task: guess the next word, check the answer, adjust. That is Next-token prediction.
The result is a base model that writes fluent text and knows a lot of facts, but does not yet follow instructions well. Later steps like Instruction tuning and RLHF (reinforcement learning from human feedback) turn it into a helpful assistant.
Years of reading everything in a giant library before anyone asks you a question. You come out knowing a lot, but nobody has yet taught you how to be a good assistant.
Believing you need to pretrain (or 'train your own model') to make a model know your company's documents. Use Retrieval-Augmented Generation (RAG) for knowledge and Fine-tuning for behaviour; pretraining is for frontier labs and a few well-funded domain teams.
Pretraining minimises next-token cross-entropy over 10–40T tokens on thousands of accelerators for weeks to months. Compute ≈ 6 × params × tokens FLOPs: Llama 3 405B used 3.8e25 FLOPs on 15.6T tokens and up to 16k H100s. The Training data mix (web, code, maths, multilingual, synthetic) plus filtering and deduplication shape capability more than any architecture tweak.
Operationally it is a reliability problem: learning-rate warmup and decay, BF16/FP8 mixed precision, data/tensor/pipeline parallelism, checkpoints every few hundred steps, and automated restarts after hardware faults and loss spikes. Most labs add a mid-training or annealing phase on high-quality and long-context data. The output is a base model whose Parametric knowledge stops at its Knowledge cutoff; helpfulness, refusals, tool use and reasoning style come from post-training on top.
Do not pretrain. Start from an Open-weights vs closed models instruct checkpoint (Llama 3.x, Qwen3, Gemma 3, Mistral) or a closed API. Consider continued pretraining only with ≥ 1B tokens of genuinely new domain text, a held-out perplexity target, and a general-capability eval to catch forgetting.
Budget for a 70B dense model on 15T tokens: 6 × 70e9 × 15e12 = 6.3e24 FLOPs. H100 BF16 dense peak is ~989 TFLOP/s; at 40% model-FLOPs utilisation that is ~400 TFLOP/s, so 1.6e10 GPU-seconds ≈ 4.4M GPU-hours — about 6,000 GPUs for a month, or ~$13M at an assumed $3/GPU-hour, before failed runs and ablations. A financial-filings team that wants the model to 'know' 50,000 10-K filings needs Retrieval-Augmented Generation (RAG), not this.
The question is build vs adapt. Say: 'I would not pretrain. I'd start from an open-weights base or instruct model, use RAG for knowledge, and fine-tune for behaviour; continued pretraining only if we have billions of tokens of domain text the base model has never seen.' Expect the follow-up 'when is continued pretraining worth it?': a new language or a heavy domain vocabulary (chemistry, a proprietary DSL) with ≥ 1B tokens.
| Run | Params | Tokens | FLOPs |
|---|---|---|---|
| Chinchilla (2022) | 70B | 1.4T | ~5.9e23 |
| Llama 3 8B (2024) | 8B | 15T | ~7.2e23 |
| Llama 3 405B (2024) | 405B | 15.6T | 3.8e25 |
Numbers worth knowing
- Pretraining compute
- ≈ 6 × params × tokens FLOPsdense transformer
- Llama 3 405B
- 3.8e25 FLOPs, 15.6T tokens, up to 16k H100sMeta, 2024
- Realistic H100 utilisation
- 35–45% MFUlarge BF16 runs
How to decide
| When | Do this |
|---|---|
| The model lacks your company's facts | Retrieval-Augmented Generation (RAG) over your documents; no training. |
| The model lacks your format, tone or task behaviour | SFT with LoRA & parameter-efficient fine-tuning on an instruct model. |
| The model lacks a whole language or domain vocabulary and you have ≥ 1B tokens | Continued pretraining from a base checkpoint, then re-run post-training. |
Cost maths
GPU-hours = 6·N·D / (peak FLOP/s × MFU × 3,600). Continued pretraining of an 8B model on 5B domain tokens: 6 × 8e9 × 5e9 = 2.4e20 FLOPs ÷ (989e12 × 0.4) ≈ 6.1e5 s ≈ 170 GPU-hours ≈ $510 at $3/GPU-hour (illustrative) — cheap in compute; the real cost is data cleaning, evals, and redoing post-training.
Operating it
- Domain perplexity of the best open base model stays > 2× its general-web perplexity after RAG and SFT
- You hold ≥ 1B tokens of clean, licensed domain text the base model has not seen
- SFT alone cannot teach a vocabulary (tokenizer splits domain terms into 5+ pieces)
- Training loss > 1.5× its 1,000-step moving average → automatic pause + checkpoint rollback
- Gradient norm > 10× rolling median
- Model-FLOPs utilisation < 30% for an hour (stragglers, failing nodes)
When it goes wrong
Training loss spiked 3× mid-run and did not recover; downstream evals from the next checkpoint collapsed.
Loss-curve alerting on the training dashboard and a jump in gradient norm two steps before the spike.
A corrupted data shard of repeated boilerplate met a learning rate at its peak; Meta's public OPT-175B logbook (2022) documents dozens of manual restarts for exactly this class of spike and hardware failure.
Roll back to the last good checkpoint, skip the offending batches, lower peak learning rate, and add per-shard data-quality checks before data enters the loader.
Related terms
Sources
- Training Compute-Optimal Large Language Models — Hoffmann et al., 2022paper
- The Llama 3 Herd of Models — Llama Team, Meta, 2024paper
- OPT: Open Pre-trained Transformer Language Models — Zhang et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.