Preference optimization (DPO)
Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.
Preference optimization teaches a model to prefer good answers over worse ones using pairs: for the same question, here is a 'chosen' answer and a 'rejected' one. The model is adjusted to make the chosen answer more likely and the rejected one less likely.
The best-known method, DPO (Direct Preference Optimization), does what RLHF (reinforcement learning from human feedback) does in a simpler, more stable way, so it is the usual choice for teams doing their own Fine-tuning.
Teaching by 'this, not that': instead of hiring a judge to score every essay and coaching toward high scores, you show the student pairs of essays and say which one is better.
Assuming DPO removes the need for good data or a reference model. Easy, obviously bad rejected answers teach almost nothing, and over-training drifts the model from the reference and erodes general skills.
DPO (Rafailov et al., 2023) shows that the KL-constrained RLHF objective has a closed-form optimal policy, so you can train with a logistic loss on pairs: maximise σ(β·[log π(y_w)/π_ref(y_w) − log π(y_l)/π_ref(y_l)]). A frozen reference model (the SFT checkpoint) plays the KL anchor; there is no reward model and no sampling during training. Typical β is 0.05–0.5 (0.1 is the common default) and learning rates are 10–100× lower than SFT.
Variants fix specific weaknesses: KTO trains on unpaired thumbs-up/down labels; SimPO and ORPO drop the reference model and length-normalise; IPO resists over-fitting to deterministic preferences; online/iterative DPO regenerates pairs from the current policy to reduce distribution shift. The known failure is length exploitation: if chosen answers are systematically longer, the model learns 'longer is better', and length-biased judges reward it (Dubois et al., 2024). Hard negatives — rejected answers that are plausible but wrong — teach far more than obviously bad ones.
DPO with LoRA (TRL or similar), β = 0.1, lr 5e-6, 1–2 epochs, reference log-probs precomputed, pairs built from real user or expert edits, length-balanced; evaluate with a length-controlled Pairwise comparison plus a general-capability regression set.
Legal-contract redlining assistant: 8,000 clauses where lawyers edited the model's suggested redline. Pairs: chosen = lawyer-edited, rejected = original draft. DPO with LoRA on an 8B instruct model, β = 0.1, lr 5e-6, 2 epochs, ~4 hours on one H100 (two forward passes per pair plus precomputed reference log-probs). On 300 held-out clauses, lawyers prefer the tuned model 64% of the time; mean length changes by +3%, confirming the gain is not verbosity.
Contrast DPO with PPO: you lose online exploration and gain simplicity, stability and one GPU. Then show you know the traps: 'I build pairs from real edits so rejected answers are hard negatives, I length-balance the pairs, and I report length-controlled win rates.' Expect: 'what if you only have thumbs up/down?' — KTO, or pair them by prompt.
RLHF (PPO)
- Train reward model
- Sample online
- 4 models in memory
- Unstable, many knobs
DPO
- No reward model
- Fixed preference pairs
- One supervised loss
- Cheap, stable, offline
| Method | Needs | Reference model | Use when |
|---|---|---|---|
| DPO | Pairs | Yes | Default |
| KTO | Thumbs up/down | Yes | No paired data |
| SimPO / ORPO | Pairs | No | Memory-bound, length drift |
| Online DPO | Pairs from policy | Yes | Offline DPO plateaus |
Numbers worth knowing
- DPO β
- 0.1 default (0.05–0.5)higher = stay closer to reference
- Compute per pair
- ~2× an SFT examplechosen + rejected forward/backward; reference log-probs cached
- Useful pair count
- 2k–20kLoRA on an instruct model
How to decide
| When | Do this |
|---|---|
| You have edited drafts or A/B choices | DPO on (edited, original) pairs. |
| You only have unpaired thumbs up/down | KTO. |
| Win rate rises but so does mean length > 10% | Length-balance pairs or switch to a length-normalised loss (SimPO). |
Cost maths
DPO GPU-hours ≈ 2 × SFT GPU-hours for the same token count. 8,000 pairs × 1,200 tokens × 2 sides × 2 epochs ≈ 38M tokens on an 8B model ≈ 4 H100-hours ≈ $12 at $3/GPU-hour (illustrative). Lawyer edits at an assumed 3 minutes × $200/hour = $10 per pair — $80k for the set. The pairs are the investment.
Operating it
- Offline DPO win rate plateaus across two data refreshes → iterative/online DPO with fresh policy samples
- General-capability regression set drops > 2 points → raise β or reduce epochs
- Pair volume from production > 50k/month → automate via a Data flywheel
- Mean output length vs previous model > +10%
- Length-controlled win rate vs raw win rate gap > 10 points
- General-skill regression suite drop > 2 points
When it goes wrong
Each monthly DPO round raised the LLM-judge win rate, yet support agents complained that replies were getting longer and slower to read.
Mean output length had grown 45% over three rounds; a human blind review showed no preference gain once length was controlled.
Agents' edited replies (chosen) were systematically longer than drafts (rejected), and the LLM-as-judge favoured length.
Length-balanced the pair set, switched reporting to length-controlled win rate, added a length budget to the release gate, and retrained with SimPO's length-normalised reward.
Related terms
Sources
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., 2023paper
- KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh et al., 2024paper
- SimPO: Simple Preference Optimization with a Reference-Free Reward — Meng et al., 2024paper
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators — Dubois et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.