Graceful degradation & load shedding
When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.
Graceful degradation means planning what your product does when something breaks. If the main model provider is down, maybe a smaller model answers; if that fails, show cached answers; if retrieval works but generation doesn't, show the matching documents with links. The user gets less, but not nothing. The opposite is a hard failure: one dependency goes down and the whole feature shows an error.
When the power fails, a hospital switches to generators that run the essential wards, not the gift shop. Less service, but the important things keep working.
A fallback that has never served real traffic. The first time it runs is during an outage, when it turns out its prompt format, context limit or rate limit doesn't fit, and you have two outages instead of one.
Design a degradation ladder per feature, ordered by quality: primary model → Model fallback & routing (another provider or a smaller self-hosted model) → cached or precomputed answer → retrieval-only (ranked sources, no generation) → static guidance with a support link. Trigger steps automatically from circuit breakers and timeouts, and expose the active rung via a Feature flag so on-call can force a step. Each rung needs its own eval: a fallback model you never tested is a second outage. Degrade along the axis users care least about: drop reranking (latency) before dropping retrieval (correctness), shorten context before switching to a weaker model for high-stakes tasks. Not everything should degrade: authorization, PII filtering and safety guardrails must Fail closed, so a missing guardrail is a stop, not a rung. Tell the user when they're on a degraded path ('showing related articles while the assistant is unavailable'), and record the rung on every trace so you can measure time spent degraded against the SLO & SLA.
Ladder defined per feature in config; circuit breaker per provider (open at 50% errors over 20 calls or p95 >2× SLO); fallback provider through an LLM gateway (LiteLLM, Portkey, or in-house); 1–5% of traffic routed to the fallback continuously to keep it warm and evaluated; rung recorded on traces; user-visible degraded banner.
A customer-support assistant has a ladder: (1) primary hosted model, (2) a second provider's comparable model via the LLM gateway, (3) a self-hosted 8B model answering only top-200 FAQ intents, (4) retrieval-only with the top 5 help-center articles. During a 70-minute provider incident, 61% of traffic was served by rung 2, 22% by rung 3, 17% by rung 4; deflection rate fell from 48% to 39% instead of 0%, and no request returned an error page.
For every LLM feature I'd define the ladder up front, eval each rung offline, trigger it from circuit breakers, and keep security controls out of the ladder entirely because those fail closed. Expect: 'when is degrading worse than failing?' When the weaker answer can cause harm, like dosage or legal advice; there the right rung is 'retrieval-only with sources' or a clear error.
Numbers worth knowing
- Keep-warm traffic to fallback
- 1–5%proves the fallback works and keeps its eval current
- Breaker open threshold
- ≥50% errors over 20 callsor latency >2× SLO for 1 min
- Quality cost per rung
- measure; typically a few points per stepon your golden set, not on public benchmarks
How to decide
| When | Do this |
|---|---|
| Primary provider erroring or slow | Switch to evaluated fallback provider via the gateway. |
| Both providers down, low-risk intent | Small self-hosted model or cached answer. |
| High-stakes intent (medical, legal, financial) | Retrieval-only with sources or a clear error, never a weaker model guessing. |
| Guardrail, auth or PII filter unavailable | Fail closed; do not degrade around security. |
Cost maths
Keeping a fallback warm costs its share of traffic: 3% of 2M requests/month at $0.012 = ~$720/month, plus a gateway. Compare to downtime: at 48% deflection and $6 per human-handled ticket, a 70-minute outage at 1,500 questions/hour that degrades to 39% instead of 0% saves ~1,500 × 70/60 × 0.39 × $6 ≈ $4,100 in ticket cost.
Operating it
- Provider incidents >2 per quarter affecting your SLO
- Time on degraded rungs >0.5% of the month
- Enterprise contracts with availability SLAs: fallback provider becomes mandatory
- Share of traffic per rung: alert when anything below rung 1 exceeds 5% for 10 min
- Fallback rung eval score (daily, golden set): alert on drop >3 points
- Circuit-breaker state per provider: page if open >5 min
When it goes wrong
During a provider outage, a fallback model returned truncated, malformed JSON and the claims UI crashed for every user.
Frontend error spike seconds after the circuit breaker opened.
The fallback had a smaller max output and didn't support the structured-output mode the primary used; it had never received production traffic.
Fallback routed 2% of traffic permanently with its own schema-validation eval, structured decoding enforced at the gateway, and a schema check on every rung's response.
Where it shows up
Sources
- Site Reliability Engineering, ch. 22: Addressing Cascading Failures — Google, 2016book
- Using load shedding to avoid overload — AWS Builders' Library, 2019blog
Check yourself
Three questions. Every option is explained once you answer.