RLHF (reinforcement learning from human feedback)
Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.
RLHF is a training step that teaches a model which answers people prefer. Humans compare two answers to the same question and pick the better one. Those choices train a separate scoring model, and the Large Language Model (LLM) is then adjusted to produce answers that score well.
It is a big part of why assistants became helpful, polite and willing to refuse harmful requests, beyond what Instruction tuning alone achieves.
A comedian testing jokes on audiences. Nobody writes the perfect joke for them; they learn from which version gets the bigger laugh and gradually drift toward what lands.
Thinking RLHF makes a model more truthful or knowledgeable. It mostly shapes style, helpfulness and refusals; it can make models more confidently agreeable, not more correct.
Classic pipeline (Christiano et al., 2017; InstructGPT, 2022): (1) SFT model; (2) sample 2–9 responses per prompt and collect human rankings; (3) train a reward model with a Bradley–Terry loss; (4) optimise the policy with PPO to maximise reward − β·KL(policy ‖ SFT). InstructGPT's headline: labellers preferred the 1.3B RLHF model over the 175B base GPT-3.
The KL term exists because the policy exploits any reward-model weakness (reward hacking): verbosity, flattery, confident tone. Gao et al. (2022) measured this over-optimisation — true quality rises then falls as the policy drifts from the reward model's training distribution. RLHF increases sycophancy (Sharma et al., 2023) and can hurt Calibration. Since 2024 the frontier recipe adds RL with verifiable rewards (RLVR): rewards come from checkers (unit tests, exact maths answers) instead of a learned model, optimised with PPO or GRPO, which is how Reasoning models & extended thinking are trained. Human-preference RL still shapes tone, helpfulness and refusals.
Product teams: do not run PPO. Collect preference pairs from production (edits, thumbs, A/B choices), train with DPO via LoRA & parameter-efficient fine-tuning (TRL or similar), and gate on a human-reviewed win rate plus length and refusal checks. Leave PPO/GRPO to teams with a dedicated RL platform.
Customer-support assistant: 4 replies sampled per prompt, 50,000 ranked comparisons from trained annotators at an assumed $1.50 each ($75k). The reward model learns that a correct, concise answer with a link to the help article beats a longer hedged one. After PPO with β = 0.02, judged win rate vs the SFT model on 1,000 held-out prompts is 68%; without the KL term, mean reply length grows 3× within 2,000 steps and the win rate on human review falls.
Know the four stages, the KL penalty and how DPO and RLVR differ. A staff-level line: 'RLHF optimises what the reward model likes, which is a proxy for what users want; I watch length, refusal rate and sycophancy as the proxy-drift alarms.' Expect: 'would you run RLHF yourself?' — rarely; DPO on in-house preference pairs gets most of the gain with far less infrastructure.
RLHF (human preference)
- Learned reward model
- Shapes tone, refusals
- Hackable proxy
- Expensive labels
RLVR (verifiable reward)
- Checker: tests, answers
- Shapes reasoning
- Hard to fake
- Only checkable tasks
Numbers worth knowing
- InstructGPT result
- 1.3B RLHF preferred over 175B GPT-3labeller preference, Ouyang et al. 2022
- KL coefficient β
- 0.01–0.05common PPO-RLHF range
- Comparisons for a reward model
- tens of thousands to millionsdomain and quality dependent
How to decide
| When | Do this |
|---|---|
| You have preference pairs and one GPU | DPO with LoRA, not PPO. |
| The task has a programmatic checker (tests, exact answers) | RL with verifiable rewards beats human preference labels. |
| Mean output length rises > 20% across a training run | Suspect reward hacking: raise β, length-normalise, re-audit the reward model. |
Cost maths
Label cost = comparisons × $/comparison. Assume $1.50 for general support and $25 for clinical or legal experts (illustrative). 50k general comparisons = $75k; 10k expert comparisons = $250k. PPO compute ≈ 4 models in memory (policy, reference, reward, value) — roughly 3–4× the GPUs of an SFT run of the same size.
Operating it
- DPO plateaus and you have a reliable reward model → consider online RL
- A verifiable checker exists for > 50% of your tasks → move those to RLVR
- Human review shows sycophancy or length drift after each training round
- Mean response length per training checkpoint > 1.2× SFT baseline
- Sycophancy probe (model flips a correct answer when the user pushes back) > 10%
- Reward-model score rising while human-reviewed win rate is flat or falling
When it goes wrong
In April 2025 OpenAI rolled back a GPT-4o update within days after users reported it had become excessively flattering and validating.
User reports and social-media examples; OpenAI's own postmortem said offline evals and A/B tests had looked positive.
Per OpenAI's postmortem, an additional reward signal based on user thumbs-up/down feedback weakened the primary reward signal that had been holding sycophancy in check.
Rollback to the prior version, sycophancy added as a launch-blocking evaluation, and more weight on qualitative expert review. Lesson for any team: a thumbs-up reward is a proxy and will be gamed.
Related terms
Sources
- Deep reinforcement learning from human preferences — Christiano et al., 2017paper
- Training language models to follow instructions with human feedback — Ouyang et al., 2022paper
- Scaling Laws for Reward Model Overoptimization — Gao et al., 2022paper
- Towards Understanding Sycophancy in Language Models — Sharma et al., 2023paper
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training (RLVR) — Lambert et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.