How do you handle rate limits and fallbacks across multiple model providers?
- 2 min — what limits exist (RPM, ITPM, OTPM) and error types
- 4 min — throttle, retry, fail over: parameters for each
- 3 min — fallback ladder and its eval requirement
- 3 min — batch separation, regional constraints, ownership
Providers enforce requests per minute and tokens per minute (Anthropic splits input and output TPM), and return 429 or 529/503 overload errors when exceeded. Resilience is three layers. Throttle first: a client- or gateway-side token bucket per key that reserves max_tokens so you rarely hit the provider's limit. Retry carefully: at most 2 retries, full-jitter backoff (base 0.5 s, cap 8 s), always honour retry-after, and a retry budget of ≤10% of requests per client so retries cannot multiply an outage. Fail over deliberately: a circuit breaker per provider-region, then a fallback ladder whose every rung has its own eval results.
Customer-support chat: interactive traffic goes to the primary model; on breaker-open it moves to the same model on a second cloud (e.g. via Bedrock or Vertex), then to a different model with its own tested prompt variant, then to degraded mode (retrieval-only answers with links). Nightly ticket summarisation runs through a queue with backpressure on the batch API at ~50% discount, so it never competes with chat for the same TPM.
A fallback model is a different product: same prompt, different refusal rate, verbosity and tool-call formatting. Each rung needs its own golden-set results and synthetic traffic every hour so it doesn't rot. Breaker settings to start from: open at >50% errors over the last 20 requests in 10 s, half-open after 30 s with 5 probe requests. Make retries safe with idempotency keys when calls trigger tool calls or side effects. Fallback rate is a first-class metric — alert at >2% for 10 minutes — because a quiet rise means you are serving a different model to users without a decision. Ownership: the gateway team owns throttling, retries and breakers centrally; feature teams own which fallback rungs are acceptable for their feature and sign off on the evals.
Support chat at 20 QPS peak × (3k in + 400 out) ≈ 68k tokens/s ≈ 4.1M TPM against an assumed 2M input-TPM tier → over the limit by 2× at peak. Options: split across two accounts or regions, cache the 2.5k-token system prompt (cached reads are cheaper and, on some providers, count differently toward limits — check current docs), or move 30% of non-interactive load to the batch API.
What the answer sounds like at each level
Retry with exponential backoff on 429s and switch to another provider if the first one fails.
Throttle with token buckets, retry with jitter under a retry budget, circuit-break per provider-region, and fall back along an evaluated ladder; move batch work to queues and batch APIs.
I centralise throttling and breakers in the gateway, require eval sign-off per fallback rung from feature owners, keep fallbacks warm with synthetic traffic, and alert on fallback rate as a product-quality signal.
Follow-ups the interviewer will push
1Retries during an outage triple your traffic. What went wrong?
No retry budget and no jitter: every client retried in lockstep. Cap retries at ≤10% of requests per client, use full jitter, honour retry-after, and let the breaker shed load instead of retrying into it.
2Can you fail over for a regulated EU tenant?
Only to routes that keep data in-region under the same DPA. If none exists, the ladder for that tenant ends at degraded mode, not at a US-hosted model.
Red flags interviewers listen for
- Unbounded retries or retries without jitter.
- Fallback to an untested model with the same prompt.
- No metric for how often fallback is serving traffic.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Retry hygiene | Jitter, caps, retry-after, retry budget, idempotency for side effects. |
| Fallback quality | Every rung evaluated, kept warm, and monitored by fallback rate. |
| Capacity separation | Moves batch work off interactive quota via queues and batch APIs. |
Sources
- Exponential Backoff and Jitter — Marc Brooker (AWS), 2015blog
- Site Reliability Engineering, ch. 21: Handling Overload — Google, 2016book
- Rate limits — Anthropic, 2025docs