A provider announces deprecation of the model you use. What is your upgrade playbook?
- 2 min — why a newer model is not drop-in
- 4 min — inventory, per-feature evals, failure-mode triage
- 3 min — shadow, canary, fallback, deadline management
- 3 min — ownership across teams and leaving reusable assets
Hosted models are retired on published schedules (Anthropic and OpenAI both maintain deprecation pages with retirement dates). A newer model is not a drop-in replacement: instruction-following, verbosity, refusal behaviour, tool-call formatting, tokenisation (so token counts and costs) and latency all shift. Model deprecation is therefore a migration with a fixed deadline, run through the same eval and rollout machinery as any risky change.
Inventory every call site and prompt version pinned to the old snapshot from gateway logs — not from a registry, which will be incomplete. Run each feature's golden set on the candidate with existing prompts, do error analysis on regressions grouped by failure mode (format drift, length, refusals), adapt prompts per mode, then shadow production traffic and ramp via canary with the old model as fallback until its retirement date.
Start the week the notice lands: prompt rework and eval cycles take 4–8 weeks for a product with a dozen features. Pin explicit snapshots org-wide so upgrades happen on your schedule. Re-check judge calibration if the judge model is also changing — otherwise you can't tell product regressions from judge shifts. Budget for cost and latency changes (new tokenisers can change token counts by 10%+ for the same text). Ownership: the platform team runs the inventory, the shared harness and the deadline; feature teams own their eval sets and sign off. Where it breaks: downstream regex parsers depending on exact formats, forgotten call sites in cron jobs, and teams who 'wait for the next model' past the deadline.
12 features × (2 days to build an eval set + 3 days of prompt rework for the ~1 in 3 that regress + 2 days shadow/canary) ≈ 12 × 3 + 4 × 3 ≈ 48 engineer-days, ~10 calendar weeks with parallel teams. Eval compute: 740 items × 2 models × 3 repeats × $0.025 ≈ $110 per full comparison.
What the answer sounds like at each level
Test the new model with our prompts, fix what breaks, and switch before the old one is turned off.
Inventory from gateway logs, run per-feature golden sets, triage regressions by failure mode, shadow, then canary with the old model as fallback.
I run deprecations as a platform programme with a deadline, a shared harness and per-team sign-off, pin snapshots org-wide, and leave behind eval sets that make the next upgrade a 2-week exercise.
Follow-ups the interviewer will push
1Two weeks before retirement, one feature still regresses 5 points. What now?
Escalate with numbers: ship the best prompt so far with an explicit owner sign-off, route its traffic to the same model on another cloud if the retirement is provider-specific, or self-host an open-weights fallback for that one feature while work continues.
2How do you avoid this fire drill next time?
Subscribe to provider deprecation feeds, keep per-feature eval sets current, and run every new candidate model through the suite within a week of release so you always know your next model before you need it.
Red flags interviewers listen for
- Swaps the model id and checks a few outputs by hand.
- Inventories call sites from a registry instead of traffic logs.
- No fallback to the old model during the ramp.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Migration rigour | Log-based inventory, per-feature evals, failure-mode triage, shadow and canary. |
| Schedule realism | Starts immediately and budgets weeks for prompt rework across features. |
| Org leadership | Runs it as a programme with clear ownership and leaves reusable eval sets. |
Migrating twelve features before a retirement date
Illustrative exampleTwelve features across six teams ran on one hosted model snapshot when the provider announced its retirement 90 days out. Prompts had been tuned to that model for a year, and there was no shared eval: each team checked quality by hand.
I led the platform side as the gateway tech lead, with three platform engineers; the twelve feature owners owned their prompts and sign-off. My goal was every feature migrated before day 75, with no user-visible quality drop and a reusable process.
I used gateway logs rather than our prompt registry to inventory call sites and found 31, four of which were in no registry (two cron jobs, two internal tools). I set up a shared harness and asked each team for a golden set of 50–80 items from logged traffic, labelled with a common rubric — 740 items in total. Offline, 8 features were within 1 point; 4 regressed: two from JSON format drift, one from answers 18% longer, and one — insurance-claim notes — where refusals rose from 0.4% to 2.1%. The finance-reporting team lead pushed back: he wanted to skip this model and wait for the next release. I showed him the retirement date was contractual, offered to keep the old model as fallback on his feature until the last week, and paired one of my engineers with his team on the prompt fix, which won him over.
All 12 features migrated by day 61, with zero user-facing incidents attributed to the migration and cost per query down 22% on the new model. When the next upgrade came, the same 740-item suite let us qualify the candidate in 9 days instead of 2 months. What I'd do differently: I let two teams skip shadow traffic to save time, and one had a downstream regex parser break at the 5% canary for 40 minutes before rollback — shadowing would have caught it for free.
Sources
- Model deprecations — Anthropic, 2025docs
- Deprecations — OpenAI, 2025docs