How do you upgrade to a new model version without regressions?
- 1 min — why upgrades regress
- 2 min — replay + diff review
- 2 min — shadow, canary, fallback
- 2 min — org calendar and waves
Treat a model upgrade like a dependency upgrade with unpredictable behaviour changes. A newer model is better on average and worse on some of your cases: formatting, refusal boundaries, verbosity, tool-calling habits, JSON fences. Even the same model name drifts when a provider updates a floating alias — measured behaviour of a hosted model changed materially across 3 months in 2023 studies. Pin dated snapshot IDs in production and upgrade through an eval and rollout process, never an alias change.
A provider announces deprecation of the current model in 6 months. The team replays the 800-item golden set plus 5,000 recent production inputs. Findings: the new model wraps JSON in markdown fences on 6% of calls, refuses 2.1% of legitimate pharmacy-dosage questions (old: 0.3%), and is 18% cheaper per task. Fixes: strip fences in the parser, add 4 counter-examples and a policy clarification to the prompt, re-run (refusals 0.4%). Then shadow 1 week, canary 5% → 25% → 100% over 2 weeks behind a feature flag.
- Pin snapshot versions; never point production at
latest. - Eval gate plus a diff review: only inspect cases where outputs changed (10–30% of items on a typical upgrade) with pairwise judging — that's where regressions hide.
- Change one thing at a time: model first with the old prompt, then prompt tweaks, then parser. Otherwise attribution is impossible.
- Keep the old model as fallback until the new one holds metrics for 2+ weeks; track drift in output length, refusal rate, tool-call error rate, cost per task.
- Org scope: the platform team runs an upgrade calendar for all deprecations and a shared harness; each product team signs off its own gate. With 20 features on one model, upgrade in waves of low-risk features first.
- Where it breaks: upgrading model, prompt and parser in one change — when something regresses you cannot tell which change caused it.
What the answer sounds like at each level
Runs the eval set on the new model before switching.
Pins versions, reviews changed outputs pairwise, stages shadow and canary, keeps fallback.
Runs an org-wide upgrade calendar and harness, sequences waves across features, and enforces one-change-at-a-time attribution.
Follow-ups the interviewer will push
1Now the old model is shut off in 2 weeks and the new one fails one feature's gate.
Ship the passing features; for the failing one, route to an alternative pinned model (another provider or an open-weights fallback) behind the gateway while fixing the prompt.
2How big must the canary be to detect a 1-point drop in task success?
At ~90% success, detecting 1 point with 80% power needs ~14.8k tasks per arm — so for low-volume features, rely on offline evals and longer shadow periods.
Red flags interviewers listen for
- Points production at a floating alias.
- Changes model, prompt and parser together.
- No fallback to the old model during rollout.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Evaluation | Golden set plus production replay, focused review of changed outputs. |
| Rollout | Shadow, staged canary with flag, fallback retained. |
| Org | Calendar, shared harness, per-team sign-off, waves. |
Sources
- How is ChatGPT's behavior changing over time? — Chen, Zaharia & Zou, 2023paper