What is the difference between offline and online evaluation, and why do you need both?
- 2 min — define each and what question each answers
- 4 min — worked example where they disagree, and why
- 4 min — the feedback loop and the offline→online hit rate
- 3 min — low-traffic and biased-signal cases, ownership
Offline evaluation runs a candidate against a fixed dataset (golden set, benchmark) before release: reproducible, minutes long, tens of dollars, zero user risk. Online evaluation measures behaviour on live traffic: task success, escalation or retry rate, conversion, A/B outcomes, plus guardrails (latency, cost, safety flags). Offline answers 'is this likely better on the cases we wrote down?'; online answers 'did users end up better off?'. Offline is the gate, online is the verdict.
A developer-docs assistant gets a new retrieval prompt. Offline, faithfulness rises 4 points (n = 500, 95% CI ±2). It ships behind a canary and a 14-day A/B on 38k sessions: 'resolved without filing a ticket' is flat, because answers got 30% longer and users skimmed past the code snippet they needed. The offline metric was real; it just wasn't the metric users paid for. Fix: add a rubric criterion 'answer leads with the runnable snippet' and a length guardrail, then re-run.
The number that tells you whether your offline suite is worth anything is the offline→online hit rate: of the last 20 changes that won offline, how many held online? Above 70% you can lean on the gate and shorten online tests; below 50% the offline set measures the wrong thing and needs error analysis against online losers. Online signals are sparse and biased — assume explicit thumbs feedback on 1–2% of sessions, skewed to the very happy and the very angry — so pair them with sampled human review and implicit signals (rephrase within 60 s, copy events, escalation). Ownership: the platform team owns the metric definitions and the experiment framework; feature teams own the mapping from their offline criteria to one online metric each. Where it breaks: an offline set that has drifted from production, and online results read before they reach significance.
Offline: 500 items × 3 repeats × $0.025 ≈ $38 and ~12 min per run. Online: baseline resolution 40%, minimum detectable effect 2 points absolute, α = 0.05, 80% power → n ≈ 15.7 × 0.24 / 0.02² ≈ 9.4k sessions per arm, ~19k total; at 2k eligible sessions/day that is ~10 days, rounded up to 14 to cover two weekly cycles.
What the answer sounds like at each level
Offline uses a test set before release; online measures real users after release. You need both because test sets miss things.
Offline gates releases on a stratified set; online A/B tests confirm on a pre-registered primary metric with guardrails, and online failures feed back into the offline set.
I track the offline→online hit rate as a health metric for the eval program itself, own one metric hierarchy across teams, and use the hit rate to decide how much online testing each change class needs.
Follow-ups the interviewer will push
1Traffic is only 300 sessions a day. Can you still do online evaluation?
Not A/B to significance on a 2-point effect — that needs ~9k sessions per arm. Instead run shadow comparisons judged offline, sample 50 sessions a week for human review, and lean harder on the offline gate with a bigger held-out slice.
2Offline and online disagree. Which do you trust?
Online, for the decision — then treat the disagreement as a bug in the offline suite: pull the sessions where the new variant lost, label them, find the missing criterion or slice, and add it before the next change.
Red flags interviewers listen for
- Treats thumbs-up rate as ground truth without discussing selection bias.
- Ships on offline wins alone with no online confirmation or guardrails.
- Has no mechanism for online failures to flow back into the offline set.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Precision of definitions | Separates gate (offline) from verdict (online) and names a primary metric plus guardrails. |
| Measurement of the measurer | Proposes measuring offline/online correlation over past launches with a threshold. |
| Signal skepticism | Quantifies feedback sparsity and bias, and proposes implicit signals plus human sampling. |
Sources
- Trustworthy Online Controlled Experiments — Kohavi, Tang & Xu, 2020book
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper