When would you use pairwise comparison instead of absolute scoring?
- 2 min — define both and the question each answers
- 4 min — worked example with interval on win rate
- 3 min — limits: no floor, quadratic cost, position bias
- 2 min — how the two combine in one harness
Absolute scoring rates each output alone (binary pass/fail per criterion, or a scale). Pairwise shows a grader two outputs for the same input and asks which is better. Humans and LLM judges are more consistent at relative judgements, so pairwise detects smaller differences with fewer items. Pairwise answers 'which is better'; absolute answers 'is it good enough'. Default: absolute binary criteria for gates, pairwise win rate for choosing between candidates that both pass. Chatbot Arena scales the same idea to many models with a Bradley–Terry fit over pairwise votes.
Two prompt variants for a code-review comment generator both pass every must-pass check at 96–97%, so absolute scoring cannot separate them. Pairwise over 300 pull-request diffs, judged in both orders: 250 decisive pairs, variant B wins 58%. With n = 250 the 95% interval is roughly ±6 points (52–64%), so B is better with confidence. Ship B, and keep the absolute gate so B can never win by being the less-bad of two broken answers.
Three limits. Pairwise has no floor: two bad answers still produce a winner, so it never replaces the gate. It scales quadratically: k variants need k(k−1)/2 comparisons per item (6 variants = 15), so for more than ~4 candidates use a tournament or a Bradley–Terry fit over sampled pairs. It inherits position bias, so always swap order and score flips as ties. For trend tracking over months, absolute scores are the only stable axis — a pairwise win rate against last month's model is not comparable to one against this month's. The platform should offer both modes in one harness; what you configure per feature is which decision each mode feeds.
Win-rate interval ≈ ±1.96 × √(0.25 / n_decisive): n = 250 → ±6.2 points; n = 1,000 → ±3.1. Cost at 300 items × 2 orders × $0.0105 per judge call ≈ $6.30 per variant pair.
Absolute (binary per criterion)
- Answers 'good enough?'
- Release gates, SLAs, trends
- Comparable across months
- Needs more items to see small gains
Pairwise win rate
- Answers 'which is better?'
- Choosing between passing candidates
- More sensitive per item
- No floor; k(k−1)/2 cost; position bias
What the answer sounds like at each level
Pairwise compares two outputs directly, which is easier to judge than giving a score.
Use pairwise with order swapping and a confidence interval on win rate to pick between candidates; keep absolute binary gates for must-pass criteria.
I standardise the split: absolute gates for release, pairwise for selection, Bradley–Terry for many-way comparisons, absolute for long-term trends — one harness, with the mode chosen by the decision being made.
Follow-ups the interviewer will push
1You have 8 prompt variants. How do you pick one?
Absolute-gate them first to drop failures, then run a Bradley–Terry fit over randomly sampled pairs (~3–5 comparisons per item) rather than all 28 pairs, and confirm the top two head-to-head on a held-out slice.
2A 52% win rate on 100 decisive pairs. Ship?
No evidence of improvement — the interval is roughly ±10 points and includes 50%. Decide on cost or latency, or collect ~1,000 decisive pairs if a 2-point win matters.
Red flags interviewers listen for
- Uses pairwise win rate as a release gate with no absolute floor.
- Reports a win rate without ties, order swapping or an interval.
- Runs all-pairs comparisons across many variants without noting the quadratic cost.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Decision mapping | Pairwise for selection, absolute for gates and trends, stated explicitly. |
| Statistics | Puts a confidence interval on win rate and handles ties and order. |
| Scaling | Knows the k(k−1)/2 cost and proposes Bradley–Terry or tournament sampling. |
Sources
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al., 2024paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper