Exponential backoff
Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.
When a call fails because a service is busy, retrying immediately makes things worse. Exponential backoff waits longer after each failure: 1 second, then 2, then 4, up to a cap. Jitter adds randomness so thousands of clients don't all retry at the same instant. LLM APIs return 429 (rate limited) and 529 or 503 (overloaded) all the time, so every client you write needs this, along with a limit on total attempts and an idempotent request.
If a shop door is jammed with people, you don't push harder every second. You step back, wait a bit longer each time, and so does everyone else, at slightly different moments.
Backoff without jitter. Every client that failed at the same moment retries at the same moment, so the service sees synchronized spikes at 1 s, 2 s, 4 s and never recovers.
The formula AWS recommends is full jitter: sleep = random(0, min(cap, base × 2^attempt)). Brooker's 2015 simulation showed full jitter completes the same work with far fewer total calls than no-jitter or equal-jitter backoff, because it spreads retries evenly instead of in synchronized waves. Always honour a server's Retry-After header over your own schedule; Anthropic and OpenAI both send it on 429s. Cap attempts (3–5) and cap total time to the caller's deadline. Retries multiply across layers: if the UI, the gateway and the SDK each retry 3 times, one user click can become 27 provider calls, so retry at exactly one layer. Add a retry budget (Google SRE: retries ≤10% of requests per client) or a circuit breaker so a hard outage doesn't triple your load. Retry only retryable errors: 408, 429, 500, 502, 503, 504, 529 and connection resets, never 400/401/403/422.
Full jitter, base 1 s, cap 30 s, max 4 retries, bounded by the request deadline; honour Retry-After; retry at the SDK layer only (disable UI and gateway retries); circuit breaker opens after 50% errors over 20 requests; retry budget 10% of requests.
A customer-support summariser calls a model API with base 1 s, cap 30 s, 5 attempts, full jitter. During a 90-second provider brownout, attempt delays are drawn from [0,1], [0,2], [0,4], [0,8], [0,16] s, so 1,200 clients spread their retries across ~30 s instead of firing 1,200 requests every second. Requests whose user-facing deadline (20 s) would be exceeded stop retrying and fall back to a smaller model.
I'd say: full-jitter exponential backoff, honour Retry-After, 3–5 attempts bounded by the request deadline, retry at one layer only, and a 10% retry budget so an outage doesn't become a self-inflicted DDoS. The probe they'll make: 'what about non-idempotent calls?' Only retry them with an idempotency key; otherwise surface the error.
| Status | Retry? | Why |
|---|---|---|
| 429 | Yes, honour Retry-After | rate limited |
| 500/502/503/504 | Yes, backoff | transient server side |
| 529 | Yes, backoff | Anthropic: overloaded |
| 400/422 | No | request is wrong |
| 401/403 | No | auth won't fix itself |
Numbers worth knowing
- Calls from layered retries
- 3 × 3 × 3 = 27three layers each retrying 3 times for one user action
- Retry budget
- ≤10% of requestsGoogle SRE per-client guidance
- Mean wait with full jitter
- ½ of the exponential ceilinguniform draw on [0, ceiling]
How to decide
| When | Do this |
|---|---|
| Server sends Retry-After | Wait at least that long; ignore your own schedule for that attempt. |
| Interactive request with a 20 s deadline | 2–3 retries max, then Model fallback & routing. |
| Batch job with hours of slack | Up to 8 retries, cap 5 min, then Dead-letter queue (DLQ). |
| Error rate >50% for 30 s | Open the circuit breaker; stop retrying and degrade. |
Cost maths
Retry amplification costs tokens when the model ran before the failure (streaming cut, gateway timeout). Assume 3k in + 400 out at $3/$15 per M (illustrative Oct 2026) = $0.015 per call. A 27× layered-retry storm on 10k failing requests = 270k calls ≈ $4,050, versus $600 with 4 retries at one layer.
Operating it
- Retry share of provider calls >10% for an hour
- Multiple providers or regions available: add model/region fallback instead of more retries
- Retries at more than one layer discovered in a trace
- Retries / total upstream calls: alert >10% over 15 min
- Upstream calls per user request (from traces): alert p99 >3
- Circuit-breaker open events: page if open >5 min
When it goes wrong
A 4-minute provider brownout turned into a 40-minute outage for an internal-wiki assistant.
Gateway traces showed each user request fanning into 9–27 upstream calls; provider returned 429 for 30 minutes after it recovered.
Frontend, API gateway and SDK each retried 3 times with fixed 1 s delays and no jitter, keeping the account above its rate limit.
Retries removed from frontend and gateway; SDK full jitter with Retry-After; 10% retry budget; circuit breaker with fallback model.
Where it shows up
Sources
- Exponential Backoff And Jitter — Marc Brooker, AWS Architecture Blog, 2015blog
- Timeouts, retries, and backoff with jitter — AWS Builders' Library, 2019blog
- Site Reliability Engineering, ch. 22: Addressing Cascading Failures — Google, 2016book
- API errors — Anthropic, 2025docs
Check yourself
Three questions. Every option is explained once you answer.