Model fallback & routing
Routing a request to a backup model or provider when the primary errors, times out or is overloaded; only safe if the backup has passed the same evals.
Model fallback means having a plan B: if the main model returns errors, is overloaded, or is too slow, the request goes to another copy of the same model on a different provider, or to a different model, instead of failing for the user.
It is a form of Graceful degradation & load shedding. The catch is that a different model may behave differently, so the backup must be tested with the same evals as the primary.
A hospital's backup generator: it keeps the critical machines running during an outage, and it is tested monthly, because a generator that has never been started is not a backup.
Configuring a fallback model that has never been evaluated with your prompt. The day it activates, structured outputs break or tone changes, and you find out from customers during an incident that was supposed to be contained.
Order fallbacks from most to least similar. First, the same model on another provider or region (a Claude model via the Anthropic API, Amazon Bedrock and Google Vertex AI; an open-weights model on two inference providers): behaviour is nearly identical, though snapshots, feature support (caching, tool-schema details) and limits differ. Second, a different model of the same tier, with its own prompt variant validated by the Eval gate (CI). Third, a smaller or cheaper model with a narrowed task, or a non-LLM degraded response ('search results only').
Triggers: HTTP 429 after honouring retry-after once, 5xx and provider overload (Anthropic returns 529 overloaded_error), time-to-first-token above a route-specific timeout, and malformed output after one repair attempt. Use a circuit breaker per provider and model (open after ~20% errors over 30 s or 5 consecutive failures, half-open probes every 30 s) so you stop hammering a sick provider, and a retry budget (retries ≤10% of traffic) so fallbacks do not amplify an outage. Fallback must be idempotent-safe for agents: if the primary may have executed a tool call before timing out, the fallback must not repeat it (Idempotency keys on side-effecting tools). Log which model served each response; quality metrics must be sliced by served model, or a 30% fallback day will look like an unexplained quality drop.
Fallback chains defined per alias in the gateway (LiteLLM fallbacks, Portkey configs); same model on a second provider first; per-provider circuit breakers; route-specific TTFT timeouts (e.g., 8 s chat, 30 s long documents); retry budget ≤10%; every fallback model × prompt pair in the eval gate; weekly forced-failover drill on 1% of traffic; served-model id logged on every response.
Insurance-claims summariser, primary: a Sonnet-class model on the first-party API, 8 s TTFT timeout. Fallback 1: the same model on a second cloud provider. Fallback 2: a smaller model with a shorter, validated prompt variant that outputs only the structured fields. During a 47-minute primary overload: 31% of requests served by fallback 1 (eval score within 0.5 points of primary), 4% by fallback 2 (structured-field accuracy 91% vs 95%, acceptable for the window), 0.2% user-visible errors. Without fallback, the error rate would have been ~35% for the duration.
Say: "My first fallback is the same model on another provider, my second is a different model with its own validated prompt, and every fallback path is in the eval gate and exercised weekly." Expect: "what about agents mid-task?": idempotency keys on tools, and resuming from durable state rather than restarting the loop on a new model.
Numbers worth knowing
- Circuit breaker open threshold
- ~20% errors over 30 s, or 5 consecutive failuresper provider × model; half-open probe every 30 s
- Retry budget
- ≤10% of requestsbeyond this, retries amplify the outage
- Acceptable eval gap for fallback 1
- within the CI of the primary (≈ ±2–3 pts)same model on another provider; larger gaps need a separate prompt variant
How to decide
| When | Do this |
|---|---|
| The same model is available from two providers | Make that the first fallback; it needs the least revalidation. |
| The fallback is a different model family | Maintain a dedicated prompt variant and gate it in CI like the primary. |
| The request may have triggered a side-effecting tool call | Do not blindly retry on another model; resume from durable state with idempotency keys. |
Cost maths
Fallback capacity costs little until used: a second provider account with on-demand pricing costs nothing idle. Cost during failover = fallback share × price delta. If fallback 2 costs 70% less, failover saves money; if fallback 1 is a provider without your prompt-cache warm, the first minutes pay full input price instead of 0.1× cache reads, e.g. 6k cached tokens × $3 / M = $0.018 vs $0.0018 per request.
Operating it
- Primary provider has more than one user-visible incident a quarter: add a second provider for the same model.
- Fallback traffic exceeds 5% on a normal day: it is no longer a fallback; capacity-plan the primary or rebalance.
- Share of traffic served by each fallback level: alert when fallback exceeds 5% for 10 minutes.
- Quality metrics (schema validity, judge score) sliced by served model: alert on any slice below its gate threshold.
- Circuit breaker state changes per provider: notify on open.
When it goes wrong
During a provider outage, an e-commerce catalog pipeline kept running at 100% success, and 18,000 product pages were published with broken attribute JSON.
Storefront rendering errors reported by merchandisers the next morning.
The fallback model had been added in config months earlier and never evaluated; it wrapped JSON in markdown fences, which the parser accepted as a string field. Success metrics counted HTTP 200s, not valid outputs, and were not sliced by served model.
Structured decoding on all paths, the fallback prompt variant added to the eval gate, success redefined as schema-valid output, dashboards sliced by served model, and a weekly 1% failover drill.
Related terms
Where it shows up
Sources
- Timeouts, retries, and backoff with jitter — Brooker / Amazon Builders' Library, 2019blog
- LiteLLM documentation — BerriAI, 2026docs
Check yourself
Three questions. Every option is explained once you answer.