Problem 121 · Evaluation

LLM-Judge Position-Bias Check by Swapping Order

MediumAI engineering

01Problem

A pairwise judge `judge(first, second) -> "first" | "second" | "tie"` compares two answers. For each pair `(a, b)`, call it in both orders, map both verdicts back to `A`/`B`/`tie`, and record the final verdict only when the two orders agree (otherwise `tie`). Return the consistency rate, the rate at which decisive verdicts favour the first slot, a two-sided exact sign-test p-value for that first-slot rate against 0.5, and the per-pair verdicts.

02Key insight

LLM judges prefer whichever answer appears first (or second, depending on the model), and also longer answers and their own outputs. Running both orders turns that bias into something you measure instead of something that silently inflates one system's win rate. Inconsistent pairs are not noise to discard; collapsing them to ties is the conservative choice, and the consistency rate itself is the judge's quality metric. The exact binomial test tells you whether the first-slot preference is real or chance on this sample size.

03Solution

solution.pypython38 lines

from math import comb
from typing import Callable

Judge = Callable[[str, str], str]
_FWD = {"first": "A", "second": "B", "tie": "tie"}
_REV = {"first": "B", "second": "A", "tie": "tie"}


def sign_test_p(k: int, n: int) -> float:
    if n == 0:
        return 1.0
    lower = sum(comb(n, i) for i in range(0, k + 1)) / 2 ** n
    upper = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * min(lower, upper))


def position_bias_report(pairs: list[tuple[str, str]], judge: Judge) -> dict:
    consistent = first_wins = decisive = 0
    verdicts: list[str] = []
    for a, b in pairs:
        v1, v2 = judge(a, b), judge(b, a)
        m1, m2 = _FWD[v1], _REV[v2]
        if m1 == m2:
            consistent += 1
            verdicts.append(m1)
        else:
            verdicts.append("tie")
        for v in (v1, v2):
            if v != "tie":
                decisive += 1
                first_wins += v == "first"
    n = len(pairs)
    return {
        "consistency": consistent / n if n else 1.0,
        "first_position_rate": first_wins / decisive if decisive else 0.5,
        "p_value": sign_test_p(first_wins, decisive),
        "verdicts": verdicts,
    }

Try it first.

TimeO(n) judge calls (2 per pair) plus O(d^2) for the exact test over d decisive verdicts

SpaceO(n)

tests.py · 8 passing

tests.pypython17 lines

pairs = [("x" * i, "y" * (i + 3)) for i in range(20)]
def by_length(f, s):
    return "tie" if len(f) == len(s) else ("first" if len(f) > len(s) else "second")
r = position_bias_report(pairs, by_length)
assert r["consistency"] == 1.0 and r["first_position_rate"] == 0.5 and r["p_value"] == 1.0
assert r["verdicts"] == ["B"] * 20
r = position_bias_report(pairs, lambda f, s: "first")
assert r["consistency"] == 0.0 and r["verdicts"] == ["tie"] * 20
assert r["first_position_rate"] == 1.0 and r["p_value"] == 2 / 2 ** 40
def leaning(f, s):
    return "first" if abs(len(f) - len(s)) < 3 else by_length(f, s)
mixed = [("aa", "bbb"), ("a", "bbbbbb"), ("aaaa", "b")]
r = position_bias_report(mixed, leaning)
assert r["verdicts"] == ["tie", "B", "A"]
assert abs(r["consistency"] - 2 / 3) < 1e-12
assert abs(r["first_position_rate"] - 4 / 6) < 1e-12
assert sign_test_p(5, 10) == 1.0 and abs(sign_test_p(0, 5) - 2 / 32) < 1e-15

04Staff-level follow-up

Swapping fixes position bias but not length or self-preference bias; control length by comparing answers of similar length or adding a length-penalty rubric, and use a judge from a different model family than the systems under test. Calibrate the judge against a few hundred human labels and report agreement (Cohen's kappa), not just accuracy. For large evals, randomise order once per pair and check bias on a sample to halve judge cost.