Problem 122 · Evaluation

Cohen's Kappa and Quadratic-Weighted Kappa

EasyAI engineering

01Problem

Write `cohen_kappa(a, b)` for two raters' categorical labels on the same items: `(p_o - p_e) / (1 - p_e)` where `p_o` is observed agreement and `p_e` the agreement expected from each rater's label frequencies. Return 1.0 when `p_e == 1` and the raters agree perfectly. Then write `weighted_kappa(a, b, categories, weights="quadratic")` for ordinal labels, where disagreement weight is `((i - j) / (k - 1)) ** 2` (or the absolute value for `"linear"`) and kappa is `1 - observed_weighted_disagreement / expected_weighted_disagreement`.

02Key insight

Raw agreement is misleading when one label dominates: two raters who both say 'pass' 90% of the time agree 82% by chance alone, so 90% agreement is only modest. Kappa subtracts that chance baseline, which is exactly what you need when validating an LLM judge against humans on skewed eval sets. For ordinal scales (1 to 5 ratings), unweighted kappa treats 4 vs 5 as badly as 1 vs 5; quadratic weighting credits near-misses and is the standard for rubric scores.

03Solution

solution.pypython32 lines

from collections import Counter
from typing import Hashable, Sequence


def cohen_kappa(a: Sequence[Hashable], b: Sequence[Hashable]) -> float:
    if len(a) != len(b) or not a:
        raise ValueError("need equal-length, non-empty label lists")
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[label] * cb[label] for label in set(ca) | set(cb)) / (n * n)
    if pe == 1.0:
        return 1.0 if po == 1.0 else 0.0
    return (po - pe) / (1 - pe)


def weighted_kappa(a: Sequence[Hashable], b: Sequence[Hashable], categories: list[Hashable],
                   weights: str = "quadratic") -> float:
    if len(a) != len(b) or not a:
        raise ValueError("need equal-length, non-empty label lists")
    k = len(categories)
    idx = {c: i for i, c in enumerate(categories)}
    power = 2 if weights == "quadratic" else 1

    def w(i: int, j: int) -> float:
        return (abs(i - j) / (k - 1)) ** power if k > 1 else 0.0

    n = len(a)
    observed = sum(w(idx[x], idx[y]) for x, y in zip(a, b)) / n
    pa, pb = Counter(idx[x] for x in a), Counter(idx[y] for y in b)
    expected = sum(pa[i] * pb[j] * w(i, j) for i in range(k) for j in range(k)) / (n * n)
    return 1.0 if expected == 0 else 1 - observed / expected

Try it first.

TimeO(n + k^2)

SpaceO(k)

tests.py · 10 passing

tests.pypython21 lines

a = ["Y"] * 25 + ["N"] * 25
b = ["Y"] * 20 + ["N"] * 5 + ["Y"] * 10 + ["N"] * 15
assert abs(cohen_kappa(a, b) - 0.4) < 1e-12
a = ["Y"] * 90 + ["N"] * 10
b = ["Y"] * 85 + ["N"] * 5 + ["Y"] * 5 + ["N"] * 5
assert abs(cohen_kappa(a, b) - (0.9 - 0.82) / 0.18) < 1e-12
assert cohen_kappa([0, 0, 1, 1] * 25, [0, 1, 0, 1] * 25) == 0.0
assert cohen_kappa(["p", "q"], ["p", "q"]) == 1.0
assert cohen_kappa(["p", "p"], ["p", "p"]) == 1.0
r1 = [1, 2, 3, 4, 5] * 10
near = [min(5, x + 1) if i % 2 else x for i, x in enumerate(r1)]
far = [1 if i % 2 else x for i, x in enumerate(r1)]
assert weighted_kappa(r1, r1, [1, 2, 3, 4, 5]) == 1.0
assert weighted_kappa(r1, near, [1, 2, 3, 4, 5]) > cohen_kappa(r1, near)
assert weighted_kappa(r1, near, [1, 2, 3, 4, 5]) > weighted_kappa(r1, far, [1, 2, 3, 4, 5])
assert weighted_kappa(r1, near, [1, 2, 3, 4, 5], "linear") < weighted_kappa(r1, near, [1, 2, 3, 4, 5])
try:
    cohen_kappa([1], [1, 2])
    assert False
except ValueError:
    pass

04Staff-level follow-up

With more than two raters use Fleiss' kappa or Krippendorff's alpha (which also handles missing labels). Report a bootstrap CI on kappa, since on 100 items it is wide. And when the judge's kappa against humans is lower than human-human kappa, the judge is the bottleneck; when they are similar, the rubric is.