How do you monitor quality drift for an LLM feature in production?
- 2 min — sources of drift and why infra metrics miss it
- 4 min — sampling, judged metrics, proxies, alert rule
- 3 min — judge canary and version attribution
- 3 min — coverage-gap clustering, ownership, closing the loop
Quality drift is degradation over time without a code change on your side: the user mix shifts, the corpus changes, upstream data breaks, or a provider updates a model behind an alias — Chen, Zaharia & Zou (2023) measured large behaviour changes in the same hosted model name across a few months. Infrastructure observability (errors, latency) won't catch it; you need quality signals computed continuously on live traffic, tied to the model snapshot and prompt version that produced each response.
E-commerce catalog Q&A with 200k responses/day: sample 2% (4k/day) and score with the same judge criteria as offline — faithfulness, abstention rate, format validity. Track proxies alongside: thumbs-down rate, rephrase-within-60 s, escalation, empty-retrieval rate, response length. Alert when the 3-day rolling mean of a judged metric drops more than 2 points below the 28-day baseline and at least one proxy moves in the same direction. Weekly, cluster recent queries by embedding and flag clusters with no golden set coverage — new product lines show up here first.
Each signal is weak alone — judges have their own noise, user feedback is sparse and biased — so require agreement across two, then confirm with ~50 sampled human reviews before acting. Monitor the monitor: re-judge a fixed 200-item judge canary set daily; if its score moves more than 2 points with no input change, the judge drifted, not the product. Log model snapshot and prompt version on every response so you can separate 'our change' from 'their change' in minutes. Ownership: the platform provides sampling, judging and alerting as a service; each feature team owns its thresholds and on-call response; every confirmed drift incident ends with new eval cases and a postmortem.
200k responses/day × 2% = 4k judged/day × (2.5k in + 200 out) = 10M in ($30) + 0.8M out ($12) ≈ $42/day at $3/$15 per M. Sensitivity: daily faithfulness 0.92 on n = 4,000 → SE ≈ √(0.92 × 0.08 / 4,000) ≈ 0.43 points, so a 2-point drop is ~4.6 SE — detectable within one day.
What the answer sounds like at each level
Watch user feedback and sample some responses to check quality over time.
Judge a fixed sample of live traffic daily with offline criteria, track proxy signals, alert on sustained shifts vs a rolling baseline, and cluster queries to find coverage gaps.
I run drift monitoring as a platform service with multi-signal alerts, a judge canary set, version attribution on every response, and a closed loop from incidents back into the eval suite.
Follow-ups the interviewer will push
1Faithfulness dropped 3 points overnight but nothing deployed. How do you triage?
Check the judge canary first (judge drift?), then group the drop by model snapshot, prompt version and query cluster. If it's one cluster, look for a corpus or catalog change; if it's everywhere on one snapshot, suspect a provider-side change and pin or fail over.
2Your traffic is 2k responses/day. Does this still work?
Judge 25–50% instead of 2% (the cost is trivial at that volume), widen the window to 7 days, and lean more on weekly human review — small samples can't detect 2-point shifts daily.
Red flags interviewers listen for
- Monitors only latency and error rates for an LLM feature.
- Single-signal alerts that fire constantly or never.
- No attribution of responses to model snapshot and prompt version.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Signal design | Judged metrics on a sized sample plus proxies, with a multi-signal alert rule. |
| Instrument integrity | Monitors the judge with a canary set and attributes responses to versions. |
| Closed loop | Confirmed incidents become eval cases and postmortems; coverage gaps feed the golden set. |
Sources
- How is ChatGPT's behavior changing over time? — Chen, Zaharia & Zou, 2023paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper