Pairwise comparison
Grading by asking which of two answers is better, not scoring each alone; more reliable for subjective quality, but needs order swaps to cancel position bias.
In pairwise comparison a grader (a person or an LLM-as-judge) sees two answers to the same question and picks the better one, or calls a tie. People and models are better at relative judgements ('A is clearer than B') than at absolute scores ('this is a 7').
It is how public leaderboards like Chatbot Arena rank models, and how teams compare a new prompt against the current one.
An optometrist asking 'better with lens one or lens two?' instead of asking you to rate your vision from 1 to 10.
Running each pair in only one order. A judge with a 10–15 point position bias can manufacture a 'win' for whichever variant you happened to put first.
Each comparison yields win/loss/tie; across many prompts you report a win rate with a CI, or fit a Bradley–Terry model (Elo-style) when ranking more than two systems. Zheng et al. (2023) found strong judges agree with human preferences over 80% of the time, about the same as human–human agreement, but documented three biases: position (preferring the first or second slot), verbosity (preferring longer answers) and self-enhancement (preferring their own model family's outputs).
Controls: run every pair in both orders and count a win only if it holds in both (else tie); instruct the judge that length is not quality and spot-check by length bucket; use a judge from a different model family than either contestant. Pairwise tells you which is better, not whether either is good enough, so pair it with absolute pass/fail checks on hard requirements. It also scales poorly for many variants: N systems need N(N−1)/2 pairings unless you use a reference baseline.
Judge from a third model family, both orders, ties allowed, explicit 'length is not quality' instruction, 300–500 prompts, report win rate excluding ties with a Wilson CI, plus a verbosity check by length bucket. Keep absolute binary checks for hard requirements alongside.
E-commerce catalog descriptions: current prompt (A) vs rewrite (B) on 400 products, judged both orders. Raw result B wins 61%. With order-swap consistency: B 49%, A 33%, inconsistent → tie 18%. Excluding the 72 ties, B wins 196 of 328 decisive pairs = 60% (95% CI ≈ 54–65%). A length-bucket check shows B's advantage is 70% on products where B is >30% longer and 51% where lengths match: half the 'win' was verbosity.
Say: "For subjective quality I compare pairwise against the production baseline, both orders, with a judge from a third model family, and I report the win rate with a CI excluding ties." Expect: "what are the judge's biases?": name position, verbosity and self-preference, and how each is controlled.
Numbers worth knowing
- Strong judge agreement with human preference
- >80%GPT-4 on MT-Bench pairs, Zheng et al. 2023; similar to human–human agreement
- Tie or inconsistent rate after order swap
- 10–25%typical when variants are close; much lower means one variant is clearly better
- Prompts to detect a 55% vs 50% win rate
- ≈ 800 decisive pairs80% power, α=0.05, two-sided
How to decide
| When | Do this |
|---|---|
| Quality is subjective (tone, clarity, helpfulness) | Use pairwise against the current production output rather than absolute scores. |
| There is a hard requirement (correct figure, valid JSON, cited clause) | Grade it absolutely with a binary check; pairwise cannot tell you both answers are wrong. |
| Ranking more than 3 variants | Compare each against one fixed baseline, or fit Bradley–Terry on a sampled pair schedule. |
Cost maths
Cost = prompts × 2 orders × judge tokens. 400 prompts × 2 × (3k in + 200 out) on a Sonnet 4.6-class judge at $3 / $15 per M ≈ 800 × $0.012 = $9.60; Batch API halves it. Order swapping doubles cost and is not optional.
Operating it
- More than 3 variants per experiment: move to a Bradley–Terry ranking with a fixed baseline.
- Win rates hover at 50–55% for several launches: increase prompts or move the decision to an online A/B test.
- Position-consistency rate (same verdict in both orders): alert below 75%.
- Win rate by length bucket: alert when the gap between long and length-matched buckets exceeds 10 points.
When it goes wrong
A customer-support team shipped a 'clearly better' prompt (judge win rate 68%); escalations to human agents rose 5% the next week.
Escalation dashboard during the rollout, then a re-run of the pairwise eval in reversed order: the win rate fell to 52%.
Pairs were judged only with the candidate in slot A, and the judge preferred slot A; the candidate was also 40% longer, which the judge rewarded.
Order swap on every pair, ties for inconsistent verdicts, a verbosity check by length bucket, and a binary 'resolves the stated issue' criterion alongside the preference.
Related terms
Where it shows up
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al., 2024paper
- Large Language Models are not Fair Evaluators — Wang et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.