Training data
The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.
Training data is everything a model learned from. For Pretraining, that is trillions of tokens of web pages, books and code. For later steps like Instruction tuning, it is smaller sets of carefully written question-and-answer examples and human preferences.
A model can only know what was in its data, up to its Knowledge cutoff, and it picks up the data's biases and mistakes too.
A cook's training is the dishes they tasted and the recipes they practised. They can improvise, but their cooking always reflects what they were exposed to.
Assuming more data is always better. Duplicates, mislabelled examples and off-distribution samples actively hurt; curation beats volume, especially for fine-tuning.
Pretraining corpora start from Common Crawl plus curated sources (code, papers, books, maths) and go through language ID, heuristic and model-based quality filters, exact and near-duplicate removal (MinHash), PII and toxicity filtering, and domain reweighting. FineWeb built 15T tokens from 96 Common Crawl snapshots this way; most raw text is discarded. Deduplication alone measurably reduces memorisation and improves perplexity (Lee et al., 2021). Since 2024, curated Synthetic data from stronger models is a large share of maths, code and reasoning data.
Post-training data differs in kind: supervised demonstrations, preference pairs for RLHF (reinforcement learning from human feedback) and Preference optimization (DPO), and verifiable problems (answers, unit tests) for RL. LIMA showed 1,000 carefully chosen SFT examples can produce a strong assistant on a good base. For builders the practical issues are Benchmark contamination (eval items leaking into training), licensing and consent, PII, and the fact that your private knowledge was never in the data — the reason for Retrieval-Augmented Generation (RAG).
For fine-tuning: 1k–10k examples sampled from real traffic, MinHash near-dedup, PII redaction, expert-corrected outputs, split by source document or customer (never by row), and a 13-gram overlap check against every eval set before training.
Insurance-claims summariser fine-tune. Start: 40,000 adjuster notes. Remove exact and near-duplicates (−18,000), notes with PII the redactor could not clean (−3,500), and summaries the QA team marked wrong (−12,000): 6,500 remain. The model tuned on 6,500 scores 81% on the Golden set; the one tuned on all 40,000 scores 72%, because duplicated templates and wrong summaries taught the wrong behaviour.
Strong answer to 'how would you build the fine-tuning set?': sample from real traffic, dedupe, redact, have experts fix rather than write from scratch, hold out by source document, and check n-gram overlap against every eval set. Expect the follow-up 'how do you know a benchmark isn't contaminated?': 13-gram overlap checks, canary strings, and a freshly written held-out set as the tiebreaker.
Numbers worth knowing
- FineWeb
- 15T tokens from 96 Common Crawl snapshotsPenedo et al., 2024
- LIMA SFT set
- 1,000 exampleson a strong 65B base model
- Llama 3 pretraining
- ~15T tokensMeta, 2024
How to decide
| When | Do this |
|---|---|
| You have > 10k noisy examples | Clean to the best 2–5k before training more; measure both. |
| Eval scores look too good | Run n-gram overlap against training data and test on a freshly written set. |
| Data contains customer PII | Redact before it enters any training or few-shot set; record consent and licence per source. |
Cost maths
Expert correction cost = examples × minutes each × hourly rate. Assume 5,000 claims summaries × 4 minutes × $60/hour (illustrative adjuster rate) = $20,000 — larger than the GPU bill for a LoRA & parameter-efficient fine-tuning fine-tune (~10 GPU-hours ≈ $30). Budget data first.
Operating it
- Golden-set gains flatten as you add data → switch from volume to error-targeted examples
- Production error clusters not represented in training data → mine them via a Data flywheel
- Train/eval n-gram overlap > 1% of eval items
- Train/eval 13-gram overlap > 1% of eval items → block training run
- Gap between offline eval and production audit accuracy > 5 points
- PII detector hits in the training set > 0
When it goes wrong
A fine-tuned claims model scored 92% offline and 74% on the first month of production claims.
A freshly written 300-item held-out set, built by people who never saw the training data, scored 75% — matching production.
Synthetic training examples were generated from the same source documents as the eval set, so the eval measured recall of those documents.
Split all data by source document, added a 13-gram overlap gate between training and eval sets, and inserted canary strings into eval files to detect future leaks.
Related terms
Sources
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — Penedo et al., 2024paper
- Deduplicating Training Data Makes Language Models Better — Lee et al., 2021paper
- LIMA: Less Is More for Alignment — Zhou et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.