Offline vs online evaluation
Offline eval scores a fixed test set before release; online eval measures real users after it. Offline is the gate, online is the verdict.
Offline evaluation runs your AI system on a saved set of test inputs (a Golden set) before any user sees the change. It is fast, cheap and repeatable.
Online evaluation measures what real users do in production: whether they had to ask again, whether the case was escalated to a human, whether they accepted the suggested code. It is the ground truth but slower and noisier. Offline tells you a change is safe to try; online tells you whether it helped.
A flight simulator versus a real flight. The simulator catches most problems cheaply and safely; only real flights show how the plane handles actual weather and actual pilots.
Optimising the offline score or thumbs-up rate until it stops meaning anything. OpenAI's April 2025 GPT-4o update passed its offline evals and small A/B tests yet was rolled back within days for sycophancy, partly because a thumbs-based reward signal rewarded agreeable answers. A metric users can please is not a quality metric.
Offline evals hold inputs fixed, which enables paired comparison and regression testing through an Eval harness and Eval gate (CI). Their weakness is representativeness: the set ages as traffic shifts (Drift), and offline scores such as judge-rated quality or Faithfulness are proxies whose correlation with outcomes must be measured, not assumed.
Online signals come in three grades. Behavioural signals (reformulation within 2 minutes, copy or accept rate, escalation to a human, task completion) are dense and hard to fake. Explicit feedback (thumbs) arrives on 0.5–3% of turns and skews toward unhappy users. Sampled online grading runs the offline judge on 1–5% of live traces, which catches Drift before outcome metrics move. Ship with Shadow traffic or a Canary & ramped rollout first, then an A/B test for the decision. Every quarter, check proxy validity: across the last 10 launches, did offline deltas predict the sign of the online delta?
Offline: 200–500 case golden set, paired runs, per-slice CIs, in CI. Online: one pre-registered decision metric (behavioural, e.g. resolved without escalation), 2–3 guardrail metrics (latency p95, cost per query, complaint rate), and an LLM judge sampling 2% of production traces through Langfuse, Arize Phoenix or Braintrust online scoring.
Insurance-claims triage assistant. Offline: a new Reranker raises judge-scored answer quality on 300 cases from 3.6 to 3.9 / 5 (paired, CI excludes 0). Online A/B, 2 weeks, 41k sessions per arm: thumbs-up 61.2% → 61.6% (inside noise; thumbs fired on only 2% of turns) but 'claim re-opened within 48 h' falls 9.1% → 7.4% (p < 0.01). The offline gain was real; it showed up in the behavioural metric, not the one the team had planned to report.
Say: "Offline is my regression gate, online is my decision metric, and I track whether the first predicts the second; if it stops predicting, I fix the eval set before trusting it." Expect the follow-up "what online metric would you pick?": name a behavioural one defined before launch, plus a guardrail metric that must not get worse.
Offline
- Fixed inputs, paired
- Minutes, ~$2–15 a run
- Proxy metrics
- Gates every PR
- Goes stale
Online
- Live traffic
- Days to weeks
- Outcome metrics
- Decides launches
- Noisy, biased feedback
Numbers worth knowing
- Explicit feedback rate
- 0.5–3% of turnsthumbs buttons in chat UIs; skewed toward negative experiences
- Online judge sample
- 1–5% of tracesenough for ±3 pt weekly CIs at 10k+ traces/day
- Offline→online sign agreement worth trusting
- ≥8 of 10 recent launchescheck quarterly; below that, rebuild the golden set from recent traffic
How to decide
| When | Do this |
|---|---|
| A change is risky (new model, new retrieval) and traffic is high | Shadow it on live inputs for 24–72 h and grade offline before exposing a single user. |
| The offline gain is significant but small (<3 pts) | Run an A/B with a pre-registered behavioural metric; do not ship on offline alone. |
| Offline and online disagree on sign twice in a quarter | Re-sample the golden set from the last 30 days of traffic and re-validate the judge. |
Cost maths
Online judging cost = traces/day × sample rate × judge tokens × price. 200k traces/day × 2% × (2k in + 150 out) on a Haiku-class judge at $1 / $5 per M ≈ 4,000 × $0.00275 ≈ $11/day. Offline full suite on the same judge ≈ $2 per run. The expensive part is the A/B duration: two weeks of a 50/50 split on a model that costs 40% more per query.
Operating it
- Traffic exceeds ~5k sessions/day: run proper A/B tests instead of before/after comparisons.
- More than 3 launches a month: automate the offline→online sign-agreement report.
- Users span several languages or regions: slice both offline and online metrics by locale.
- Decision metric per arm with a 95% CI, daily during experiments.
- Online judge pass rate on sampled traces: alert on a 3-point drop week over week.
- Golden-set staleness: share of cases older than 90 days; alert above 60%.
When it goes wrong
A developer-docs assistant's offline score rose 6 points after a prompt rewrite, yet support tickets tagged 'docs bot wrong' rose 30% the following week.
Weekly ticket-tag dashboard, then sampled online judging that showed failures concentrated on a new SDK version released two weeks earlier.
The golden set was built from traffic six months old and contained no questions about the new SDK; the rewrite improved old-version answers and degraded new-version ones.
Re-sampled 40% of the golden set from the last 30 days, added a 'current release' slice, and made the online judge sample stratified by SDK version.
Sources
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Kohavi, Tang & Xu, 2020book
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- A postmortem of three recent issues — Anthropic, 2025postmortem
Check yourself
Three questions. Every option is explained once you answer.