Model deprecation & upgrades
A provider retiring a model snapshot on a set date; every dependent prompt, eval and fallback must be migrated and re-validated before requests start failing.
Model providers regularly retire older models. They announce a deprecation, name a replacement and set a retirement date, after which requests to the old model fail. Anthropic, for example, gives at least 60 days' notice for publicly released models.
A migration is not a find-and-replace of the model name: the new model behaves differently, so prompts, output parsing, Guardrails and costs all need re-checking through your evals.
A supplier discontinuing a component in your product: a successor part exists, but it has different dimensions, so you re-test the assembly before the old stock runs out.
Swapping the model id a week before retirement because 'the new one is better on benchmarks'. Benchmarks don't test your parser, your prompt or your judge, and a week leaves no time for a canary.
Plan for it as a recurring event, not a surprise. Inventory: which routes, prompts, fallbacks, batch jobs and eval judges use which snapshot (the LLM gateway logs make this a query; provider usage exports help). Migrate in stages: run the candidate through the Eval gate (CI) with the current prompt, fix regressions with a new prompt version, shadow real traffic for 1–3 days comparing outputs, Canary & ramped rollout at 5–25%, then cut over well before retirement.
What breaks: changed defaults and parameters (Anthropic's Claude 4.7+ models return a 400 for non-default temperature, top_p or top_k, so code that set temperature 0 for 'determinism' fails outright), output-format drift that breaks parsers, different refusal and verbosity behaviour, tool-call style changes, tokeniser changes that move cost and context fit, and the LLM judge itself if it was the deprecated model, which silently shifts every eval baseline. Chen, Zaharia & Zou (2023) showed that even the same model name can change behaviour between versions: GPT-4's accuracy on one prime-identification task moved from 84% to 51% between March and June 2023. Pin snapshots, keep the deprecation calendar in your planning, and keep migration capacity budgeted each quarter.
Pinned snapshots in prompt bundles; a deprecation calendar fed by provider deprecation pages and email notices; a gateway query listing usage by snapshot; a standing migration checklist: gate → prompt fix → judge re-validation → shadow → canary → cut-over, finishing ≥3 weeks before retirement; quarterly capacity reserved for migrations; open-weights fallback for routes that cannot tolerate forced changes.
Anthropic deprecated claude-sonnet-4-5-20250929 on 30 Sep 2026 with retirement on 30 Nov 2026 and claude-sonnet-5-5 as the recommended replacement. A clinical-intake summariser on that snapshot: week 1, inventory finds 3 routes plus the eval judge; week 2, the gate fails 2 slices (medication lists more verbose, one parser broke on a new heading style) and every request that set temperature: 0 errors; week 3, prompt v19 fixes both, judge migrated and re-validated against nurse labels (kappa 0.71); weeks 4–5, shadow then 25% canary; cut over by week 6, three weeks before retirement.
Say: "I treat deprecations as scheduled migrations: inventory from gateway logs, gate the replacement with my evals, re-validate the judge if it's affected, shadow, canary, and finish weeks before the date." Expect: "what breaks most often?": parameters the new model rejects, output-format drift, and silently shifted eval baselines when the judge model changes too.
Numbers worth knowing
- Anthropic notice before retirement
- ≥60 dayspublicly released models, per Anthropic's deprecation policy (Oct 2026)
- Behaviour shift under one model name
- 84% → 51% on a prime-identification taskGPT-4, March vs June 2023, Chen et al.
- Typical migration effort
- 1–3 engineer-weeks per productwith an existing eval gate; much longer without one
How to decide
| When | Do this |
|---|---|
| The deprecated model is also your eval judge | Migrate and re-validate the judge against human labels first, then re-baseline all scores before comparing generators. |
| Regulatory or contractual constraints forbid unplanned behaviour changes | Run open-weights models you host, or negotiate extended availability in the provider contract. |
| The replacement is cheaper or faster | Still go through gate, shadow and canary; cost wins don't excuse unvalidated behaviour. |
Cost maths
Migration cost ≈ engineer time + eval runs + shadow traffic. 2 engineer-weeks ≈ $12k loaded; eval runs ≈ $100; shadowing 100% of 50k requests/day for 2 days at $0.02 ≈ $2k. Price changes can repay it: moving from $3 / $15 to $2 / $10 per M (Sonnet 4.6-class to Sonnet 5.5-class list, Oct 2026) cuts a $40k/month bill by about a third, if token counts stay similar; re-measure, since tokenisers and verbosity differ.
Operating it
- More than 5 routes or teams on hosted models: maintain a central inventory and migration runbook owned by the platform team.
- Two migrations in a year slipped within 2 weeks of retirement: reserve standing migration capacity each quarter.
- Requests per model snapshot vs the deprecation calendar: alert when any deprecated snapshot still has traffic 21 days before retirement.
- Calls to provider APIs bypassing the gateway: alert on any.
- Eval baseline shift after a judge migration: flag any score change above the CI as a re-baseline, not a quality change.
When it goes wrong
On retirement day, a financial-filings extraction batch job failed every request; a backlog of 12,000 documents built up over a weekend.
Batch failure alerts Monday morning; the API returned errors for the retired model id.
Interactive routes had been migrated via the gateway alias, but the batch job called the provider directly with a hard-coded snapshot id, so it was missing from the gateway-based inventory.
Blocked direct provider keys outside the gateway, added a CI lint for hard-coded model ids, and made the deprecation runbook include provider usage exports, not just gateway logs.
Related terms
Where it shows up
Sources
- Model deprecations — Anthropic, 2026docs
- How is ChatGPT's behavior changing over time? — Chen, Zaharia & Zou, 2023paper
- A postmortem of three recent issues — Anthropic, 2025postmortem
Check yourself
Three questions. Every option is explained once you answer.