Data flywheel
The loop that turns production traces and feedback into labelled failures, new eval cases and fixes, so each week of traffic makes the system measurably better.
A data flywheel is a routine, not a product: you log what your AI system does in production (Observability & traces), pick out the interactions that went wrong or look risky, have people label them, and turn them into new test cases and fixes. The next release is tested against those cases, so the same failure cannot quietly return.
Each turn of the loop makes the Golden set more realistic and the system better, which is why it is called a flywheel.
An airline's incident-reporting system: every near miss is written up, analysed and turned into a checklist item, so the whole fleet learns from one crew's bad day.
Logging everything and calling it a flywheel. Without a weekly reading session and an owner who turns failures into cases, the trace store is an expensive archive.
Five stages, each with an owner. Capture: trace every request with inputs, retrieved context, tool calls, outputs, model and prompt version, after PII redaction. Select: sample 100% of explicit negatives (thumbs-down, escalations, overrides), plus a 1–2% random sample, plus targeted pulls (low judge scores, long agent trajectories, new intents). Label: weekly Error analysis session where engineers and a domain owner read 50–100 traces and assign failure categories. Encode: each new category gets 5–20 cases and a grader in the Eval harness. Fix and gate: the fix ships through the Eval gate (CI), and the case stays forever as a regression test.
The flywheel stalls in predictable ways: traces are logged but nobody reads them; labels have no owner; privacy rules block engineers from seeing traces (Anthropic's 2025 quality postmortem cites limited access to affected conversations as one reason detection was slow); or the selection only follows thumbs-down, which under-samples silent failures. Measure the loop itself: time from a user-reported failure to a gated regression case, and the share of production failures already covered by a case.
Tracing with OpenTelemetry GenAI conventions into Langfuse, Arize Phoenix, Braintrust or LangSmith; PII redaction at capture; a saved query for thumbs-down, escalations and low judge scores; a 1–2% random sample; a 60–90 minute weekly review with a named owner; a failure taxonomy in a shared sheet; new cases committed to the golden set within the same week.
Developer-docs assistant, ~20k questions/week. Selection: 300 thumbs-down + 40 escalations + 200 random samples. Weekly 90-minute session: two engineers and a docs writer read 80 traces, tag 23 failures, 9 of them a new category ('answers for deprecated v2 API'). They add 12 cases and a grader that checks version tags. Fix: a metadata filter on doc version. Over 8 weeks the suite grows from 220 to 390 cases and the share of new thumbs-down already covered by a case rises from 31% to 68%.
Say: "The flywheel is a weekly ritual with an owner: sample negatives and randoms, read them, encode new failure modes as cases, and gate the fix. I measure time from report to regression case." Expect: "how do you handle privacy?": redact at capture, role-based access to raw traces, and retention limits.
Numbers worth knowing
- Traces read per weekly session
- 50–100two engineers plus one domain owner, 60–90 minutes
- Random sample rate
- 1–2% of trafficcatches silent failures that never get a thumbs-down
- Coverage target
- ≥60% of new user-reported failures match an existing caseafter ~2 months of weekly sessions
How to decide
| When | Do this |
|---|---|
| Thumbs-down volume is too low to sample from (<50/week) | Rely on random samples and judge-flagged traces; add behavioural signals like reformulation. |
| Privacy rules block raw trace access | Redact at capture and give the review group role-based access to redacted traces, rather than skipping review. |
| A failure category recurs after a fix | Check the case exists, is in the gated slice, and the grader actually fails the old behaviour. |
Cost maths
People dominate. Weekly session: 3 people × 1.5 h × $120/h loaded ≈ $540/week. Tracing storage: 20k traces × 15 KB ≈ 300 MB/week, negligible. Online judge on a 2% sample: 400 traces × ~$0.003 ≈ $1.20/week. The ROI case: one prevented regression in a support product typically saves more escalation cost than a year of sessions.
Operating it
- More than 500 flagged traces a week: add judge-based pre-clustering so humans read representatives, not everything.
- More than one product team shares the model: shared failure taxonomy and a cross-team weekly review.
- Report-to-case time exceeds 2 weeks: assign a rotating flywheel owner.
- Median time from flagged trace to committed regression case: alert above 14 days.
- Share of new failures covered by existing cases: alert if it falls for 3 consecutive weeks.
- Weekly traces reviewed: alert if a week passes with zero.
When it goes wrong
A customer-support assistant's thumbs-down rate was flat for six months, yet churn surveys cited 'the bot gives up' as a top complaint.
A random 1% sample review found 14% of conversations ended with the bot asking the user to 'contact support' on answerable questions.
Selection used only thumbs-down; users who were deflected simply left without rating, so the most common failure never entered the flywheel.
Added random sampling and a 'deflection on answerable question' judge to the selection, created 25 cases for that failure, and cut deflections to 5% within a month.
Related terms
Where it shows up
Sources
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper
- A postmortem of three recent issues — Anthropic, 2025postmortem
Check yourself
Three questions. Every option is explained once you answer.