Drift
Production behaviour shifting from what you evaluated: user inputs, corpus, provider model, or what 'correct' means changes under you.
A system that scored well at launch can quietly get worse because the world around it changes. Users start asking different things, documents get added or go out of date, the AI provider updates the model, or the right answer changes (a new policy, a new product). That slow slide is drift. You catch it by watching inputs and outputs over time and re-running your evals regularly.
A map that was perfect when printed; new roads open and old ones close, and the map gets worse without a single line on it changing.
Using a floating model alias and assuming nothing changed because your code did not. Provider updates are deploys you did not review.
Four kinds, each with its own detector. Input drift: query distribution shifts (new intents, languages, lengths); detect with intent-classifier mix, embedding-centroid distance, or Population Stability Index (PSI > 0.2 is a common alarm). Corpus drift: documents added, changed, or stale; detect with index freshness, canary queries, and retrieval-score distributions. Model drift: the provider changes behaviour behind an alias; Chen, Zaharia and Zou (2023) documented large swings between March and June 2023 GPT-4 snapshots on identical tasks. Detect by pinning dated snapshots and running a fixed regression set daily. Concept/label drift: the right answer changes (new regulation, price change); detect with golden-set staleness reviews and user feedback. Monitoring proxies (thumbs-down, abstention rate, average reranker score, answer length) move before quality metrics do; treat a shift in any as a trigger for a targeted eval run.
Pin dated model snapshots; daily 100-item regression run; weekly full golden set with per-slice trends; dashboards for intent mix (PSI), mean top reranker score, abstention rate, answer length and thumbs-down; index freshness and canary-query probes every 5–15 min; golden items carry an as_of date and are reviewed quarterly.
Clinical-coding assistant mapping notes to ICD-10-CM codes. October brings the annual code-set update; 300+ codes added or revised. Inputs look the same, the model is pinned, retrieval works, yet accuracy on affected chapters falls from 92% to 71%: concept drift. Detected because the golden set is tagged by code chapter and re-run weekly; fixed by re-indexing the new code set, adding effective-date filters, and refreshing 15% of golden items with updated expected codes.
Say: I name the four drifts, input, corpus, model and concept, and give each a detector: intent and embedding mix for inputs, freshness and canaries for the corpus, pinned snapshots plus a daily regression run for the model, and dated golden items for concepts. Expect: 'which alarm fires first?' Usually a proxy, such as abstention rate or mean reranker score, days before quality metrics.
Numbers worth knowing
- PSI alarm
- > 0.2intent or length distribution vs 30-day baseline; 0.1–0.2 = watch
- Regression-run size
- 100 items dailydetects ~10-point shifts; full set weekly for smaller ones
- Provider snapshot notice
- typically months before retirementdated snapshots; floating aliases can change without notice
How to decide
| When | Do this |
|---|---|
| Input drift with no quality drop | Add new intents to the golden set; no code change yet |
| Quality drop on one slice, inputs stable | Check corpus freshness and concept changes for that slice |
| Provider snapshot retiring | Full eval on the successor, side-by-side, before switching the pin |
Cost maths
Daily regression: 100 items × ~$0.02 = $2/day. Weekly full run of 500 items with judge ≈ $15. Under $150/month total to detect a silent model or corpus change that would otherwise surface through customer complaints.
Operating it
- PSI > 0.2 on intent mix for a week
- Any slice drops > 5 points in the daily run
- Provider announces deprecation of the pinned snapshot
- Intent-mix PSI vs 30-day baseline; alert > 0.2
- Daily regression accuracy per slice; alert on a > 5-point drop
- Mean top-1 reranker score and abstention rate; alert on > 15% shifts
- Index freshness lag; alert > 2× the ingestion SLA
When it goes wrong
A JSON-extraction pipeline's parse-failure rate went from 0.3% to 6% overnight with no deploy
Parse-failure alert on the ingestion queue; the dead-letter queue filled with responses wrapped in markdown code fences
The app called the provider through a floating model alias that moved to a newer snapshot with different formatting habits
Pinned a dated snapshot, switched to native structured outputs with a JSON schema, and added the daily regression run with a parse-rate check
Related terms
Where it shows up
Sources
- How is ChatGPT's behavior changing over time? — Chen, Zaharia & Zou, 2023paper
- A survey on concept drift adaptation — Gama et al., 2014paper
Check yourself
Three questions. Every option is explained once you answer.