Explain pretraining, instruction tuning and RLHF / DPO.
- 2 min — pretraining objective, scale, over-training rationale
- 2 min — SFT and preference tuning (RLHF vs DPO)
- 1 min — RLVR and reasoning models
- 2 min — what an application team should own
Pretraining: next-token prediction over trillions of tokens of text and code (Llama 3: ~15T tokens for an 8B model — far beyond the Chinchilla-optimal ~20 tokens/parameter, because over-training a small model makes it cheaper to serve). Produces capability; behaves like autocomplete. SFT: supervised training on prompt→ideal-response pairs, 10k–1M examples; teaches the assistant format. Preference tuning: RLHF trains a reward model on human comparisons and optimises the policy with PPO against it; DPO optimises directly on chosen/rejected pairs with no reward model or RL loop. RLVR (2024–2026): RL against programmatic checkers on maths and code — the recipe behind reasoning models.
A base model completes "How do I rotate an API key?" with three more FAQ questions; the chat model answers it. Capability came from pretraining; the behaviour of answering came from post-training. Practical consequence for an application team: if your fine-tune is supposed to teach new knowledge, you're fighting the wrong stage; if it's teaching format and tone, SFT on 1–5k good examples is enough.
- Most capability is set in pretraining; post-training shapes behaviour, safety and format. A small fine-tune rarely adds knowledge reliably.
- Preference tuning has known side effects: verbosity, sycophancy, over-refusal. Test for them after every tuning stage.
- DPO runs on one training pipeline with a frozen reference model — the reason application teams use it; PPO-based RLHF needs reward model + policy + value model + reference in memory, 3–4× the infrastructure.
- Org scope: almost no product org should pretrain. Buy the base model; build SFT/DPO capability only if you have ≥1k high-quality domain examples and an eval showing prompting has plateaued. A central ML platform team owns the training pipeline; product teams own data and acceptance evals.
- Where it breaks: optimising against an imperfect reward — reward hacking. The model learns to please the judge (longer answers, confident tone) rather than be correct.
What the answer sounds like at each level
Describes the three stages and what each adds.
Explains reward models vs DPO, side effects of preference tuning, and RLVR for reasoning.
Maps stages to build-vs-buy: buy pretraining, selectively own SFT/DPO with clear data and eval prerequisites, and guards against reward hacking.
Follow-ups the interviewer will push
1Why train an 8B model on 15T tokens if Chinchilla says ~160B?
Chinchilla minimises training compute for a target loss. If the model will serve billions of requests, spending more training compute on a smaller model minimises total cost, since inference dominates.
2When would you pick DPO over SFT?
When you have pairs of acceptable vs better outputs (e.g. reviewer edits) and need to shift preferences like concision or tone; SFT alone imitates, DPO contrasts.
Red flags interviewers listen for
- Believes fine-tuning is how you add a company's knowledge base.
- Cannot distinguish reward-model RLHF from DPO.
- Ignores sycophancy/verbosity side effects.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Accuracy | Correct objectives and data for each stage. |
| Trade-offs | RLHF vs DPO cost and control; preference side effects. |
| Org judgement | Clear build-vs-buy with prerequisites for owning a stage. |
Sources
- Training language models to follow instructions with human feedback — Ouyang et al., 2022paper
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., 2023paper
- Training Compute-Optimal Large Language Models — Hoffmann et al., 2022paper